Jul 29 07:15:17.153706 ip-10-0-143-226 systemd[1]: Starting Kubernetes Kubelet... Jul 29 07:15:17.607568 ip-10-0-143-226 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 07:15:17.607568 ip-10-0-143-226 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 07:15:17.607568 ip-10-0-143-226 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 07:15:17.607568 ip-10-0-143-226 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 07:15:17.607568 ip-10-0-143-226 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 07:15:17.608685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.608553 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 07:15:17.614995 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.614981 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:15:17.614995 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.614995 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615001 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615004 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615007 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615010 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615013 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615016 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615019 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615022 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615024 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615027 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615030 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615032 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:15:17.615061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615035 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615087 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615128 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615132 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615135 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615138 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615141 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615144 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615147 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615152 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615154 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615157 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615160 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615165 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615173 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615176 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615179 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615182 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615185 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:15:17.615573 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615187 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615190 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615193 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615203 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615206 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615208 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615723 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615740 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615746 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615751 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615756 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615761 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615766 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615771 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615776 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615781 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615785 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615790 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615794 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615804 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615809 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:15:17.616230 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615813 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615818 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615823 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615827 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615832 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615840 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615848 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615853 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615859 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615864 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615876 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615880 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615885 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615889 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615893 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615897 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615902 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615906 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615913 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:15:17.616791 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615917 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615921 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615930 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615941 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615946 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615950 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615954 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615959 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615963 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615968 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615972 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615976 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615981 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.615994 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616034 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616040 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616045 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616051 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616061 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:15:17.617254 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616069 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616074 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616080 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616085 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616090 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616094 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.616105 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617025 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617033 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617036 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617040 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617043 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617045 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617048 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617051 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617053 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617056 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617059 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617061 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617064 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:15:17.617737 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617066 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617069 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617071 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617074 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617076 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617079 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617082 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617084 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617088 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617091 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617093 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617096 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617098 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617101 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617103 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617106 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617108 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617111 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617113 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:15:17.618258 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617116 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617119 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617122 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617125 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617128 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617130 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617133 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617135 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617138 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617140 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617143 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617145 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617148 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617150 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617153 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617155 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617157 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617160 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617163 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617167 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:15:17.618763 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617170 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617173 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617176 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617179 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617182 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617185 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617187 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617190 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617194 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617196 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617199 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617201 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617205 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617208 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617211 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617213 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617216 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617219 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617221 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617223 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:15:17.619261 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617226 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617228 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617231 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617233 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617236 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617239 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617241 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617243 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617246 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617248 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617251 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617254 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617256 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617260 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617264 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617267 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617270 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617273 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617290 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:15:17.619754 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617294 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617299 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617302 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617305 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617308 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617310 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617313 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617316 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617698 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617704 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617706 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617709 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617712 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617715 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617718 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617720 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617723 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617726 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617728 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617731 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:15:17.620218 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617733 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617736 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617739 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617742 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617744 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617747 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617750 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617753 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617756 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617759 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617762 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617764 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617767 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617770 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617773 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617776 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617778 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617781 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617784 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617787 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:15:17.620713 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617790 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617792 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617795 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617797 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617799 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617802 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617804 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617807 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617811 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617815 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617817 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617820 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617823 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617826 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617829 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617831 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617834 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617838 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617842 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:15:17.621194 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617845 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617848 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617853 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617856 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617860 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617862 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617865 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617868 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617870 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617873 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617875 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617878 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617880 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617883 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617885 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617888 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617892 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617894 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617897 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:15:17.621682 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617899 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617901 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617904 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617907 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617911 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617914 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617916 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617919 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617922 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617924 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617927 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617929 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617932 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617935 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617938 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617940 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617943 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617945 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617948 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617950 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:15:17.622135 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617953 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617955 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617958 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617961 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617963 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617966 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617968 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617971 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.617974 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618045 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618052 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618059 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618065 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618071 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618076 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618086 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618091 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618094 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618098 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618101 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618104 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618107 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 07:15:17.622631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618110 2565 flags.go:64] FLAG: --cgroup-root="" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618113 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618116 2565 flags.go:64] FLAG: --client-ca-file="" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618120 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618123 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618128 2565 flags.go:64] FLAG: --cluster-domain="" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618131 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618135 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618138 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618141 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618144 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618147 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618150 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618153 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618156 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618159 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618162 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618165 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618168 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618172 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618175 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618178 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618181 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618184 2565 flags.go:64] FLAG: --enable-server="true" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618187 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 07:15:17.623153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618191 2565 flags.go:64] FLAG: --event-burst="100" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618196 2565 flags.go:64] FLAG: --event-qps="50" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618198 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618201 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618204 2565 flags.go:64] FLAG: --eviction-hard="" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618208 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618211 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618213 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618217 2565 flags.go:64] FLAG: --eviction-soft="" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618219 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618224 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618227 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618230 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618233 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618236 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618239 2565 flags.go:64] FLAG: --feature-gates="" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618242 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618245 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618248 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618251 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618254 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618257 2565 flags.go:64] FLAG: --help="false" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618259 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-143-226.ec2.internal" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618263 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 07:15:17.623833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618266 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618269 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618273 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618276 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618279 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618282 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618284 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618287 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618290 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618294 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618297 2565 flags.go:64] FLAG: --kube-reserved="" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618300 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618303 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618306 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618308 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618311 2565 flags.go:64] FLAG: --lock-file="" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618314 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618317 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618321 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618327 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618329 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618332 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618335 2565 flags.go:64] FLAG: --logging-format="text" Jul 29 07:15:17.624503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618338 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618341 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618344 2565 flags.go:64] FLAG: --manifest-url="" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618347 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618351 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618354 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618357 2565 flags.go:64] FLAG: --max-pods="110" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618360 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618363 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618367 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618369 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618372 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618376 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618379 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618386 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618389 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618392 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618395 2565 flags.go:64] FLAG: --pod-cidr="" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618399 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618405 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618408 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618411 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618414 2565 flags.go:64] FLAG: --port="10250" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618417 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 07:15:17.625069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618420 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c256cda778e37df6" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618424 2565 flags.go:64] FLAG: --qos-reserved="" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618442 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618447 2565 flags.go:64] FLAG: --register-node="true" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618450 2565 flags.go:64] FLAG: --register-with-taints="" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618454 2565 flags.go:64] FLAG: --registry-burst="10" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618457 2565 flags.go:64] FLAG: --registry-qps="5" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618460 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618463 2565 flags.go:64] FLAG: --reserved-memory="" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618467 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618470 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618473 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618475 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618478 2565 flags.go:64] FLAG: --runonce="false" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618481 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618484 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618487 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618489 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618492 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618495 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618498 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618502 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618504 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618507 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618510 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618513 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 07:15:17.625677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618516 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618520 2565 flags.go:64] FLAG: --system-cgroups="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618522 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618528 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618531 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618533 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618538 2565 flags.go:64] FLAG: --tls-min-version="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618541 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618543 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618548 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618551 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618553 2565 flags.go:64] FLAG: --v="2" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618557 2565 flags.go:64] FLAG: --version="false" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618561 2565 flags.go:64] FLAG: --vmodule="" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618565 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618568 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618675 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618679 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618682 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618684 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618687 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618690 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618692 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:15:17.626298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618695 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618698 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618700 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618703 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618706 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618709 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618711 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618714 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618717 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618719 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618724 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618726 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618729 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618731 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618734 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618736 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618739 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618742 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618746 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618748 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:15:17.626885 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618752 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618756 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618760 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618764 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618768 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618772 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618777 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618781 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618785 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618789 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618792 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618795 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618797 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618800 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618803 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618805 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618808 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618810 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618814 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:15:17.627379 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618816 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618819 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618821 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618825 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618828 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618830 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618833 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618836 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618838 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618841 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618844 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618848 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618850 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618853 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618855 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618858 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618860 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618863 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618865 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618868 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618870 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:15:17.627871 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618873 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618875 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618878 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618880 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618883 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618886 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618888 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618891 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618895 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618898 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618901 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618905 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618907 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618910 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618914 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618917 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618920 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618923 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618925 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:15:17.628386 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618929 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618932 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618935 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618939 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618943 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618946 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618948 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618951 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618953 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618956 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618958 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618961 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.618964 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.618969 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.625505 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 07:15:17.628910 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.625520 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.625539 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.625545 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625598 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625603 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625606 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625609 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625612 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625615 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625617 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625620 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625623 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625625 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625628 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625630 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625633 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625636 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625638 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625641 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625644 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:15:17.629279 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625646 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625650 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625654 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625657 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625660 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625662 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625665 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625668 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625670 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625672 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625675 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625677 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625682 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625685 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625688 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625690 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625693 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625696 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625698 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:15:17.629782 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625701 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625703 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625706 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625708 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625711 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625713 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625716 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625718 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625721 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625723 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625726 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625728 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625731 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625733 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625736 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625738 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625741 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625744 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625746 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:15:17.630298 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625749 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625751 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625754 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625758 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625761 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625764 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625768 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625772 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625775 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625778 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625781 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625784 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625786 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625789 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625792 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625795 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625797 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625800 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625802 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:15:17.630959 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625804 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625807 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625811 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625814 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625817 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625819 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625822 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625824 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625827 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625829 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625832 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625835 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625838 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625840 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625843 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625845 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625848 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625850 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625853 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625855 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:15:17.631487 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625858 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625861 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625864 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625866 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625869 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.625874 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625982 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625986 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625989 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625992 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625995 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.625998 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626000 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626003 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626005 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 29 07:15:17.631989 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626008 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626010 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626013 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626015 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626018 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626020 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626022 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626025 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626028 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626032 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626035 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626038 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626041 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626043 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626046 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626048 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626051 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626054 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626056 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 07:15:17.632355 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626062 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626065 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626067 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626070 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626072 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626075 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626077 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626080 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626082 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626084 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626087 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626089 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626092 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626094 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626097 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626099 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626102 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626104 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626108 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 07:15:17.632875 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626111 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626114 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626117 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626119 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626121 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626124 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626127 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626129 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626132 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626135 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626138 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626140 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626143 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626146 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626150 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626153 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626155 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626158 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626160 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 07:15:17.633351 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626163 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626165 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626168 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626171 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626173 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626175 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626178 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626180 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626183 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626186 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626189 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626191 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626194 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626196 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626199 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626202 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626205 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626207 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626210 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626212 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 07:15:17.633870 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626215 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626217 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626220 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626222 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626225 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626227 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626229 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626232 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626235 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626237 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626240 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626242 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:17.626245 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.626250 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 07:15:17.634771 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.626343 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 07:15:17.635146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.628393 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 07:15:17.635146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.629777 2565 server.go:1019] "Starting client certificate rotation" Jul 29 07:15:17.635146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.629893 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 07:15:17.635146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.630825 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 07:15:17.635146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.633052 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 07:15:17.639565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.639540 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 07:15:17.652172 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.652152 2565 log.go:25] "Validated CRI v1 runtime API" Jul 29 07:15:17.654483 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.654465 2565 log.go:25] "Validated CRI v1 image API" Jul 29 07:15:17.655266 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.655250 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 07:15:17.657592 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.657575 2565 fs.go:135] Filesystem UUIDs: map[3204d336-b80b-4b5c-acb0-6a8423b7bc43:/dev/nvme0n1p3 3e94037a-9d7b-44fa-beff-fe145da8d4f6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 29 07:15:17.657635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.657592 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 07:15:17.663618 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.663514 2565 manager.go:216] Machine: {Timestamp:2026-07-29 07:15:17.661219032 +0000 UTC m=+0.375921725 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108166 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f41980612746f61da556d3d7801e6 SystemUUID:ec2f4198-0612-746f-61da-556d3d7801e6 BootID:3e74535b-14e8-4982-9539-618a3d92a586 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c1:43:c4:24:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c1:43:c4:24:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:b7:51:43:80:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 07:15:17.663618 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.663614 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 07:15:17.663745 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.663684 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 07:15:17.667289 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.667267 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 07:15:17.667434 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.667292 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 07:15:17.668043 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.668033 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 07:15:17.668080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.668046 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 07:15:17.668080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.668058 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 07:15:17.668724 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.668714 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 07:15:17.669270 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.669252 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 07:15:17.669615 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.669603 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 07:15:17.671323 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671310 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 29 07:15:17.671457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671447 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 07:15:17.671494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671478 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 29 07:15:17.671525 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671500 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 07:15:17.671525 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671513 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 07:15:17.671525 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671524 2565 kubelet.go:398] "Adding apiserver pod source" Jul 29 07:15:17.671623 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671536 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 07:15:17.671869 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671858 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 07:15:17.671915 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.671877 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 07:15:17.674197 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.674182 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 07:15:17.676193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.676178 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 07:15:17.676277 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.676216 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 07:15:17.677823 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677808 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677828 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677838 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677845 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677857 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677866 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677878 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 07:15:17.677889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677886 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 07:15:17.678106 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677896 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 07:15:17.678106 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677904 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 07:15:17.678106 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677916 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 07:15:17.678106 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.677925 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 07:15:17.679404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.679392 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 07:15:17.679480 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.679409 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 07:15:17.681676 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.681655 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 07:15:17.681756 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.681697 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 07:15:17.682486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.682469 2565 server.go:1268] "Started kubelet" Jul 29 07:15:17.682580 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.682553 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 07:15:17.682639 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.682547 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 07:15:17.682639 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.682618 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 07:15:17.686225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.685788 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 07:15:17.686225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.686095 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 07:15:17.686546 ip-10-0-143-226 systemd[1]: Started Kubernetes Kubelet. Jul 29 07:15:17.688114 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.688038 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 07:15:17.688397 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.688365 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 29 07:15:17.695735 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.693854 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-226.ec2.internal.18c6b15d8ea84073 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-226.ec2.internal,UID:ip-10-0-143-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-226.ec2.internal,},FirstTimestamp:2026-07-29 07:15:17.682446451 +0000 UTC m=+0.397149144,LastTimestamp:2026-07-29 07:15:17.682446451 +0000 UTC m=+0.397149144,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-226.ec2.internal,}" Jul 29 07:15:17.696340 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.696327 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 07:15:17.696395 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.696332 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 07:15:17.696465 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.696418 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:17.696976 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.696945 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 07:15:17.696976 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.696947 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 07:15:17.696976 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.696983 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 07:15:17.697150 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697055 2565 factory.go:55] Registering systemd factory Jul 29 07:15:17.697150 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697071 2565 factory.go:223] Registration of the systemd container factory successfully Jul 29 07:15:17.697150 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697112 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 29 07:15:17.697150 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697123 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 29 07:15:17.697329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697298 2565 factory.go:153] Registering CRI-O factory Jul 29 07:15:17.697329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697314 2565 factory.go:223] Registration of the crio container factory successfully Jul 29 07:15:17.697419 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697361 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 07:15:17.697419 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697383 2565 factory.go:103] Registering Raw factory Jul 29 07:15:17.697419 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697397 2565 manager.go:1196] Started watching for new ooms in manager Jul 29 07:15:17.697786 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.697772 2565 manager.go:349] Starting recovery of all containers Jul 29 07:15:17.702004 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.701905 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 07:15:17.702075 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:17.702006 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 07:15:17.708013 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.707992 2565 manager.go:354] Recovery completed Jul 29 07:15:17.712676 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.712664 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:17.714544 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.714518 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:17.714626 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.714554 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:17.714626 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:17.714569 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:18.823737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.823658 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 07:15:18.823737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.823674 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 07:15:18.823737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.823690 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 29 07:15:18.823866 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.823661 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:18.824362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.824350 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:18.824427 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.824371 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:18.824427 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.824380 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:18.825566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.825551 2565 policy_none.go:49] "None policy: Start" Jul 29 07:15:18.825566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.825568 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 07:15:18.825668 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.825578 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 07:15:18.826519 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.826509 2565 policy_none.go:47] "Start" Jul 29 07:15:18.828533 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.828379 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-226.ec2.internal.18c6b15d9091e430 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-226.ec2.internal,UID:ip-10-0-143-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-226.ec2.internal,},FirstTimestamp:2026-07-29 07:15:17.714535472 +0000 UTC m=+0.429238164,LastTimestamp:2026-07-29 07:15:17.714535472 +0000 UTC m=+0.429238164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-226.ec2.internal,}" Jul 29 07:15:18.836374 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.836312 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-226.ec2.internal.18c6b15d90924a62 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-226.ec2.internal,UID:ip-10-0-143-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-226.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-226.ec2.internal,},FirstTimestamp:2026-07-29 07:15:17.714561634 +0000 UTC m=+0.429264326,LastTimestamp:2026-07-29 07:15:17.714561634 +0000 UTC m=+0.429264326,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-226.ec2.internal,}" Jul 29 07:15:18.846729 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.846673 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-226.ec2.internal.18c6b15d909294d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-226.ec2.internal,UID:ip-10-0-143-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-226.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-226.ec2.internal,},FirstTimestamp:2026-07-29 07:15:17.714580694 +0000 UTC m=+0.429283387,LastTimestamp:2026-07-29 07:15:17.714580694 +0000 UTC m=+0.429283387,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-226.ec2.internal,}" Jul 29 07:15:18.850918 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.850900 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z54l6" Jul 29 07:15:18.853523 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853512 2565 manager.go:337] "Starting Device Plugin manager" Jul 29 07:15:18.853583 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.853539 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 07:15:18.853583 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853549 2565 server.go:87] "Starting device plugin registration server" Jul 29 07:15:18.853785 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853774 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 07:15:18.853844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853787 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 07:15:18.853893 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853878 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 07:15:18.853986 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853972 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 07:15:18.853986 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.853987 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 07:15:18.854420 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.854402 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 07:15:18.854474 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.854445 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:18.859032 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.858970 2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-226.ec2.internal.18c6b15d9091e430\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-226.ec2.internal.18c6b15d9091e430 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-226.ec2.internal,UID:ip-10-0-143-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-226.ec2.internal,},FirstTimestamp:2026-07-29 07:15:17.714535472 +0000 UTC m=+0.429238164,LastTimestamp:2026-07-29 07:15:18.824360828 +0000 UTC m=+0.430538927,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-226.ec2.internal,}" Jul 29 07:15:18.862178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.862160 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z54l6" Jul 29 07:15:18.954163 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.954144 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:18.955337 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.955009 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:18.955337 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.955037 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:18.955337 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.955047 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:18.955337 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.955082 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-226.ec2.internal" Jul 29 07:15:18.965394 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:18.965377 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-226.ec2.internal" Jul 29 07:15:18.965436 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.965397 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-226.ec2.internal\": node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:18.988278 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:18.988220 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.011808 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.011785 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 07:15:19.013016 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.013002 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 07:15:19.013089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.013040 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 07:15:19.013089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.013057 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 07:15:19.013089 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.013082 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 07:15:19.015022 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.015008 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:15:19.089238 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.089221 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.113720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.113676 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal"] Jul 29 07:15:19.113806 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.113746 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:19.115333 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.115319 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:19.115416 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.115353 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:19.115416 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.115368 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:19.116667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.116653 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:19.116804 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.116789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.116855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.116816 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:19.117383 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.117369 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:19.117429 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.117397 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:19.117429 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.117412 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:19.117429 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.117422 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:19.117536 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.117444 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:19.117536 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.117474 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:19.118777 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.118764 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.118823 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.118790 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:19.119432 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.119415 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:19.119502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.119449 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:19.119502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.119464 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:19.142544 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.142524 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-226.ec2.internal\" not found" node="ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.146781 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.146768 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-226.ec2.internal\" not found" node="ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.189316 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.189301 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.207926 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.207907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfb2bb48160090119761ed3ae3f83567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal\" (UID: \"dfb2bb48160090119761ed3ae3f83567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.208025 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.207937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfb2bb48160090119761ed3ae3f83567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal\" (UID: \"dfb2bb48160090119761ed3ae3f83567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.208025 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.207978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e309e9e9463c00491d4bf68a40b67053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-226.ec2.internal\" (UID: \"e309e9e9463c00491d4bf68a40b67053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.289918 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.289853 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.309028 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.309008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e309e9e9463c00491d4bf68a40b67053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-226.ec2.internal\" (UID: \"e309e9e9463c00491d4bf68a40b67053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.309093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.309049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfb2bb48160090119761ed3ae3f83567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal\" (UID: \"dfb2bb48160090119761ed3ae3f83567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.309093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.309009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e309e9e9463c00491d4bf68a40b67053-config\") pod \"kube-apiserver-proxy-ip-10-0-143-226.ec2.internal\" (UID: \"e309e9e9463c00491d4bf68a40b67053\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.309093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.309070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfb2bb48160090119761ed3ae3f83567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal\" (UID: \"dfb2bb48160090119761ed3ae3f83567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.309202 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.309117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfb2bb48160090119761ed3ae3f83567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal\" (UID: \"dfb2bb48160090119761ed3ae3f83567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.309202 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.309139 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfb2bb48160090119761ed3ae3f83567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal\" (UID: \"dfb2bb48160090119761ed3ae3f83567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.390339 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.390317 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.445904 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.445885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.451081 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.451060 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:19.490917 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.490896 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.591442 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.591388 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.691836 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.691815 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.737210 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.737190 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 07:15:19.737563 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.737310 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 07:15:19.792696 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.792671 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.867613 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.867544 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 07:10:18 +0000 UTC" deadline="2028-03-10 08:12:29.264954755 +0000 UTC" Jul 29 07:15:19.867613 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.867576 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14160h57m9.3973823s" Jul 29 07:15:19.893710 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.893684 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.908683 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.908668 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:15:19.993931 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.993911 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 07:15:19.993931 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:19.993928 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:19.994802 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.994783 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientMemory" Jul 29 07:15:19.994886 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.994817 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 07:15:19.994886 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:19.994827 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeHasSufficientPID" Jul 29 07:15:20.094757 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.094729 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-226.ec2.internal\" not found" Jul 29 07:15:20.120822 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.120770 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:15:20.136390 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:20.136367 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb2bb48160090119761ed3ae3f83567.slice/crio-f296ee8167151bac3e0d5ca510bbc240dc1494199113c58c2c4656f61e2490b3 WatchSource:0}: Error finding container f296ee8167151bac3e0d5ca510bbc240dc1494199113c58c2c4656f61e2490b3: Status 404 returned error can't find the container with id f296ee8167151bac3e0d5ca510bbc240dc1494199113c58c2c4656f61e2490b3 Jul 29 07:15:20.136736 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:20.136717 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode309e9e9463c00491d4bf68a40b67053.slice/crio-0233cff904e8681ed5f760eb56f5546051ab691f172e2b6a6932f02c3865e4ee WatchSource:0}: Error finding container 0233cff904e8681ed5f760eb56f5546051ab691f172e2b6a6932f02c3865e4ee: Status 404 returned error can't find the container with id 0233cff904e8681ed5f760eb56f5546051ab691f172e2b6a6932f02c3865e4ee Jul 29 07:15:20.143051 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.143038 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:15:20.205558 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.205540 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" Jul 29 07:15:20.213400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.213386 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 07:15:20.215394 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.215382 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" Jul 29 07:15:20.226337 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.226324 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 07:15:20.280020 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.279997 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:15:20.781037 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.781010 2565 apiserver.go:52] "Watching apiserver" Jul 29 07:15:20.790576 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.790544 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 07:15:20.791460 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.791435 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-kfrhz","openshift-image-registry/node-ca-r4v44","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal","openshift-multus/multus-additional-cni-plugins-bp8tg","openshift-multus/multus-r9jk6","openshift-multus/network-metrics-daemon-7pqft","kube-system/global-pull-secret-syncer-vhxbk","kube-system/konnectivity-agent-sgp6l","openshift-dns/node-resolver-pqrbt","openshift-network-diagnostics/network-check-target-pj9lt","openshift-network-operator/iptables-alerter-fcjth","openshift-ovn-kubernetes/ovnkube-node-8cg7x","kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt"] Jul 29 07:15:20.793820 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.793783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.797154 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.797132 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.797390 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.797373 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.797522 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.797473 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 07:15:20.797716 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.797700 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.798024 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.798006 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 07:15:20.799088 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.799070 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m7sks\"" Jul 29 07:15:20.800214 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.800170 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x7crp\"" Jul 29 07:15:20.800722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.800403 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.800722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.800417 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 07:15:20.800722 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.800479 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:20.800722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.800522 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 07:15:20.802937 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.802636 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.803947 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.803897 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:20.803947 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.803975 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:20.805338 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.805322 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 07:15:20.805828 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.805357 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fsr7d\"" Jul 29 07:15:20.805828 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.805424 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 07:15:20.805828 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.805712 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 07:15:20.806288 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.806269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.807539 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.807519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:20.809187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.809168 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.809276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.809190 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7q5d2\"" Jul 29 07:15:20.809276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.809203 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.809795 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.809500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:20.810722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.810351 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.810722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.810470 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.810722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.810578 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pzv6s\"" Jul 29 07:15:20.810916 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.810827 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 07:15:20.811243 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.811151 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:20.811243 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.811214 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:20.812437 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.812422 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxhbp\"" Jul 29 07:15:20.812705 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.812492 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.812705 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.812606 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.812859 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.812827 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:20.814337 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.814179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.815555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815535 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 07:15:20.815693 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815674 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.815859 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-etc-kubernetes\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.815922 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815868 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.815922 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f31a0c1a-5717-41ba-a99f-b218218bbbf6-dbus\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.816057 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-system-cni-dir\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.816057 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysctl-d\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816057 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815993 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fq6fw\"" Jul 29 07:15:20.816212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.815994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-tuned\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-conf-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.816212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.816212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.816212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-systemd\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-var-lib-kubelet\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab02368-6cde-4341-b13e-379b3edd30c6-tmp\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qj4v\" (UniqueName: \"kubernetes.io/projected/8ab02368-6cde-4341-b13e-379b3edd30c6-kube-api-access-7qj4v\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c362a7ff-dfdb-429b-acb5-45b263ee3950-cni-binary-copy\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.816457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-cni-multus\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.816841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/342f3143-0c98-4b3b-b48d-a95ba0044fd7-konnectivity-ca\") pod \"konnectivity-agent-sgp6l\" (UID: \"342f3143-0c98-4b3b-b48d-a95ba0044fd7\") " pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.816841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhv2\" (UniqueName: \"kubernetes.io/projected/5e4006b4-33f1-494a-aee9-6c62a5ed904a-kube-api-access-xqhv2\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:20.816841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-modprobe-d\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.816841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/342f3143-0c98-4b3b-b48d-a95ba0044fd7-agent-certs\") pod \"konnectivity-agent-sgp6l\" (UID: \"342f3143-0c98-4b3b-b48d-a95ba0044fd7\") " pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.816841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.817132 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816857 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 07:15:20.817132 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.816888 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.817132 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-sys\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.817132 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817112 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-system-cni-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-cni-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817191 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-kubelet\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-os-release\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-socket-dir-parent\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cnibin\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.817321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbf8\" (UniqueName: \"kubernetes.io/projected/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-kube-api-access-7fbf8\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-host\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-cnibin\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-k8s-cni-cncf-io\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-netns\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-cni-bin\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817418 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-multus-certs\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f31a0c1a-5717-41ba-a99f-b218218bbbf6-kubelet-config\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysctl-conf\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-run\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-lib-modules\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-hostroot\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-daemon-config\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgtm\" (UniqueName: \"kubernetes.io/projected/c362a7ff-dfdb-429b-acb5-45b263ee3950-kube-api-access-nlgtm\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.817656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.818281 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-os-release\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.818281 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysconfig\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.818281 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.817730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-kubernetes\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.818475 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818359 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fwv4d\"" Jul 29 07:15:20.818533 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818512 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 07:15:20.818533 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818519 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.818721 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818699 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 07:15:20.819253 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818855 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 07:15:20.819253 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818916 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6984b\"" Jul 29 07:15:20.819253 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.818996 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 07:15:20.824412 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.824392 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 07:15:20.854210 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.854155 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9x7x6" Jul 29 07:15:20.863694 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.863655 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9x7x6" Jul 29 07:15:20.906624 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.906560 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 07:15:20.918675 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-sys\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.918790 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-system-cni-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.918790 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-kubelet\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.918790 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92389c6d-c9a1-471b-a501-7c646ada778e-iptables-alerter-script\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:20.918790 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-log-socket\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-system-cni-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-os-release\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-sys\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918833 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-kubelet\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-socket-dir-parent\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cnibin\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovnkube-script-lib\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d7c6a4b-d721-4564-9325-0c0aaf987eed-serviceca\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cnibin\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-multus-certs\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f31a0c1a-5717-41ba-a99f-b218218bbbf6-kubelet-config\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.919014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919017 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-socket-dir-parent\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-run\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-hostroot\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919104 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-run\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-daemon-config\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgtm\" (UniqueName: \"kubernetes.io/projected/c362a7ff-dfdb-429b-acb5-45b263ee3950-kube-api-access-nlgtm\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-os-release\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f31a0c1a-5717-41ba-a99f-b218218bbbf6-kubelet-config\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jzm\" (UniqueName: \"kubernetes.io/projected/92389c6d-c9a1-471b-a501-7c646ada778e-kube-api-access-z9jzm\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919255 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-kubelet\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-var-lib-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysconfig\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-etc-kubernetes\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-system-cni-dir\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-systemd-units\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.919635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovn-node-metrics-cert\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.918876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-os-release\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysconfig\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-multus-certs\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-etc-kubernetes\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-system-cni-dir\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919651 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-registration-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d7c6a4b-d721-4564-9325-0c0aaf987eed-host\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-tuned\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-conf-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-os-release\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919853 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-hostroot\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-etc-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.919984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-cni-netd\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jnv\" (UniqueName: \"kubernetes.io/projected/b38d9af8-cc44-431b-b3a9-59d63f50990c-kube-api-access-s7jnv\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:20.920444 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920023 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-systemd\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-conf-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-systemd\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-daemon-config\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.920147 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-cni-multus\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhv2\" (UniqueName: \"kubernetes.io/projected/5e4006b4-33f1-494a-aee9-6c62a5ed904a-kube-api-access-xqhv2\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.920276 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:15:21.420242796 +0000 UTC m=+3.026420895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-modprobe-d\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovnkube-config\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghjw\" (UniqueName: \"kubernetes.io/projected/b681c55a-d0f8-4e57-923a-c9f02c238ab7-kube-api-access-bghjw\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-cni-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-run-netns\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-cni-multus\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-node-log\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.921176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-cni-bin\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-device-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbf8\" (UniqueName: \"kubernetes.io/projected/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-kube-api-access-7fbf8\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-slash\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920731 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcchh\" (UniqueName: \"kubernetes.io/projected/db2e9088-3a0d-401b-872b-0db4b9c8315c-kube-api-access-vcchh\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-socket-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b38d9af8-cc44-431b-b3a9-59d63f50990c-hosts-file\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-host\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-cnibin\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-multus-cni-dir\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.920976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-modprobe-d\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-k8s-cni-cncf-io\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921029 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-cnibin\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-host\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-k8s-cni-cncf-io\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-netns\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-cni-bin\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.921911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921103 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-run-netns\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c362a7ff-dfdb-429b-acb5-45b263ee3950-host-var-lib-cni-bin\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysctl-conf\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-lib-modules\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.921347 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-lib-modules\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysctl-conf\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921397 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd42d\" (UniqueName: \"kubernetes.io/projected/9d7c6a4b-d721-4564-9325-0c0aaf987eed-kube-api-access-bd42d\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:20.921415 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:21.421400309 +0000 UTC m=+3.027578398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-kubernetes\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f31a0c1a-5717-41ba-a99f-b218218bbbf6-dbus\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-kubernetes\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92389c6d-c9a1-471b-a501-7c646ada778e-host-slash\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f31a0c1a-5717-41ba-a99f-b218218bbbf6-dbus\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:20.922758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921553 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-systemd\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysctl-d\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-sysctl-d\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-ovn\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921731 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-env-overrides\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-sys-fs\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-var-lib-kubelet\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab02368-6cde-4341-b13e-379b3edd30c6-tmp\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qj4v\" (UniqueName: \"kubernetes.io/projected/8ab02368-6cde-4341-b13e-379b3edd30c6-kube-api-access-7qj4v\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c362a7ff-dfdb-429b-acb5-45b263ee3950-cni-binary-copy\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab02368-6cde-4341-b13e-379b3edd30c6-var-lib-kubelet\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/342f3143-0c98-4b3b-b48d-a95ba0044fd7-konnectivity-ca\") pod \"konnectivity-agent-sgp6l\" (UID: \"342f3143-0c98-4b3b-b48d-a95ba0044fd7\") " pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.923555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38d9af8-cc44-431b-b3a9-59d63f50990c-tmp-dir\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/342f3143-0c98-4b3b-b48d-a95ba0044fd7-agent-certs\") pod \"konnectivity-agent-sgp6l\" (UID: \"342f3143-0c98-4b3b-b48d-a95ba0044fd7\") " pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.921989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-run-ovn-kubernetes\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.922063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cni-binary-copy\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.922094 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.922393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c362a7ff-dfdb-429b-acb5-45b263ee3950-cni-binary-copy\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.922579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/342f3143-0c98-4b3b-b48d-a95ba0044fd7-konnectivity-ca\") pod \"konnectivity-agent-sgp6l\" (UID: \"342f3143-0c98-4b3b-b48d-a95ba0044fd7\") " pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.923864 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab02368-6cde-4341-b13e-379b3edd30c6-etc-tuned\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.924387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.924182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab02368-6cde-4341-b13e-379b3edd30c6-tmp\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.925974 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.925939 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/342f3143-0c98-4b3b-b48d-a95ba0044fd7-agent-certs\") pod \"konnectivity-agent-sgp6l\" (UID: \"342f3143-0c98-4b3b-b48d-a95ba0044fd7\") " pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:20.932845 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.932818 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgtm\" (UniqueName: \"kubernetes.io/projected/c362a7ff-dfdb-429b-acb5-45b263ee3950-kube-api-access-nlgtm\") pod \"multus-r9jk6\" (UID: \"c362a7ff-dfdb-429b-acb5-45b263ee3950\") " pod="openshift-multus/multus-r9jk6" Jul 29 07:15:20.933738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.933717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbf8\" (UniqueName: \"kubernetes.io/projected/aa243776-81a0-49b9-b1fa-3dc5083aaf4b-kube-api-access-7fbf8\") pod \"multus-additional-cni-plugins-bp8tg\" (UID: \"aa243776-81a0-49b9-b1fa-3dc5083aaf4b\") " pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:20.934660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.934639 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhv2\" (UniqueName: \"kubernetes.io/projected/5e4006b4-33f1-494a-aee9-6c62a5ed904a-kube-api-access-xqhv2\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:20.934874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.934834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qj4v\" (UniqueName: \"kubernetes.io/projected/8ab02368-6cde-4341-b13e-379b3edd30c6-kube-api-access-7qj4v\") pod \"tuned-kfrhz\" (UID: \"8ab02368-6cde-4341-b13e-379b3edd30c6\") " pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:20.985468 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:20.985446 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 07:15:21.017367 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.017319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" event={"ID":"dfb2bb48160090119761ed3ae3f83567","Type":"ContainerStarted","Data":"f296ee8167151bac3e0d5ca510bbc240dc1494199113c58c2c4656f61e2490b3"} Jul 29 07:15:21.017792 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.017767 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" event={"ID":"e309e9e9463c00491d4bf68a40b67053","Type":"ContainerStarted","Data":"0233cff904e8681ed5f760eb56f5546051ab691f172e2b6a6932f02c3865e4ee"} Jul 29 07:15:21.023188 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jzm\" (UniqueName: \"kubernetes.io/projected/92389c6d-c9a1-471b-a501-7c646ada778e-kube-api-access-z9jzm\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.023297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023191 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-kubelet\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-var-lib-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-systemd-units\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023283 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-kubelet\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovn-node-metrics-cert\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-registration-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023373 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023387 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-registration-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-var-lib-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-systemd-units\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d7c6a4b-d721-4564-9325-0c0aaf987eed-host\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.023874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d7c6a4b-d721-4564-9325-0c0aaf987eed-host\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.023874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:21.023874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-etc-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023764 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-etc-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-cni-netd\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.023874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-cni-netd\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jnv\" (UniqueName: \"kubernetes.io/projected/b38d9af8-cc44-431b-b3a9-59d63f50990c-kube-api-access-s7jnv\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovnkube-config\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.023981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghjw\" (UniqueName: \"kubernetes.io/projected/b681c55a-d0f8-4e57-923a-c9f02c238ab7-kube-api-access-bghjw\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-run-netns\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-node-log\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-cni-bin\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-device-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-slash\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcchh\" (UniqueName: \"kubernetes.io/projected/db2e9088-3a0d-401b-872b-0db4b9c8315c-kube-api-access-vcchh\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-socket-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b38d9af8-cc44-431b-b3a9-59d63f50990c-hosts-file\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-node-log\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd42d\" (UniqueName: \"kubernetes.io/projected/9d7c6a4b-d721-4564-9325-0c0aaf987eed-kube-api-access-bd42d\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92389c6d-c9a1-471b-a501-7c646ada778e-host-slash\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-systemd\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-ovn\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-env-overrides\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-sys-fs\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38d9af8-cc44-431b-b3a9-59d63f50990c-tmp-dir\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-run-ovn-kubernetes\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92389c6d-c9a1-471b-a501-7c646ada778e-iptables-alerter-script\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-log-socket\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovnkube-script-lib\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.024640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d7c6a4b-d721-4564-9325-0c0aaf987eed-serviceca\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024655 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-systemd\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-sys-fs\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-cni-bin\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024727 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovnkube-config\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-openvswitch\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-device-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-run-ovn\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-slash\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.024974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-run-netns\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d7c6a4b-d721-4564-9325-0c0aaf987eed-serviceca\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-socket-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025174 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b38d9af8-cc44-431b-b3a9-59d63f50990c-hosts-file\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-log-socket\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b681c55a-d0f8-4e57-923a-c9f02c238ab7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92389c6d-c9a1-471b-a501-7c646ada778e-host-slash\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db2e9088-3a0d-401b-872b-0db4b9c8315c-host-run-ovn-kubernetes\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.025494 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92389c6d-c9a1-471b-a501-7c646ada778e-iptables-alerter-script\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.026326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025714 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38d9af8-cc44-431b-b3a9-59d63f50990c-tmp-dir\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.026326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025749 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovnkube-script-lib\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.026326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.025778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db2e9088-3a0d-401b-872b-0db4b9c8315c-env-overrides\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.026326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.026050 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db2e9088-3a0d-401b-872b-0db4b9c8315c-ovn-node-metrics-cert\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.038381 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.038334 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:21.038381 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.038355 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:21.038381 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.038365 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:21.038582 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.038427 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:21.53841208 +0000 UTC m=+3.144590184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:21.040606 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.040585 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jzm\" (UniqueName: \"kubernetes.io/projected/92389c6d-c9a1-471b-a501-7c646ada778e-kube-api-access-z9jzm\") pod \"iptables-alerter-fcjth\" (UID: \"92389c6d-c9a1-471b-a501-7c646ada778e\") " pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.041050 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.041031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jnv\" (UniqueName: \"kubernetes.io/projected/b38d9af8-cc44-431b-b3a9-59d63f50990c-kube-api-access-s7jnv\") pod \"node-resolver-pqrbt\" (UID: \"b38d9af8-cc44-431b-b3a9-59d63f50990c\") " pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.046771 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.046746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghjw\" (UniqueName: \"kubernetes.io/projected/b681c55a-d0f8-4e57-923a-c9f02c238ab7-kube-api-access-bghjw\") pod \"aws-ebs-csi-driver-node-8hsqt\" (UID: \"b681c55a-d0f8-4e57-923a-c9f02c238ab7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.047331 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.047312 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd42d\" (UniqueName: \"kubernetes.io/projected/9d7c6a4b-d721-4564-9325-0c0aaf987eed-kube-api-access-bd42d\") pod \"node-ca-r4v44\" (UID: \"9d7c6a4b-d721-4564-9325-0c0aaf987eed\") " pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.047486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.047467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcchh\" (UniqueName: \"kubernetes.io/projected/db2e9088-3a0d-401b-872b-0db4b9c8315c-kube-api-access-vcchh\") pod \"ovnkube-node-8cg7x\" (UID: \"db2e9088-3a0d-401b-872b-0db4b9c8315c\") " pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.109250 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.109227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r9jk6" Jul 29 07:15:21.118750 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.118730 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:21.128670 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.128652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" Jul 29 07:15:21.134449 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.134432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" Jul 29 07:15:21.145005 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.144986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r4v44" Jul 29 07:15:21.153647 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.153630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pqrbt" Jul 29 07:15:21.161347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.161328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcjth" Jul 29 07:15:21.169106 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.169088 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:21.174698 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.174681 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" Jul 29 07:15:21.427401 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.427325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:21.427401 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.427382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:21.427618 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.427479 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:21.427618 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.427546 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:15:22.42753134 +0000 UTC m=+4.033709428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:21.427618 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.427488 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:21.427757 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.427628 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:22.427614067 +0000 UTC m=+4.033792155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:21.628719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.628685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:21.628882 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.628863 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:21.628974 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.628893 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:21.628974 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.628906 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:21.629082 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:21.628983 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:22.628947599 +0000 UTC m=+4.235125700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:21.867169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.867085 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 07:10:20 +0000 UTC" deadline="2028-01-10 11:58:10.054627488 +0000 UTC" Jul 29 07:15:21.867169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:21.867125 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12724h42m48.187508379s" Jul 29 07:15:21.967608 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.967501 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc362a7ff_dfdb_429b_acb5_45b263ee3950.slice/crio-b0828e3a689b12ba84fa6d1cb43b9962e2b777da6b1215ae3210e02d2f9b752f WatchSource:0}: Error finding container b0828e3a689b12ba84fa6d1cb43b9962e2b777da6b1215ae3210e02d2f9b752f: Status 404 returned error can't find the container with id b0828e3a689b12ba84fa6d1cb43b9962e2b777da6b1215ae3210e02d2f9b752f Jul 29 07:15:21.971711 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.971688 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f3143_0c98_4b3b_b48d_a95ba0044fd7.slice/crio-647b70b53df377c9dfe56ef7ae9c2233355ca3063be6785d20a65a175c4f30a4 WatchSource:0}: Error finding container 647b70b53df377c9dfe56ef7ae9c2233355ca3063be6785d20a65a175c4f30a4: Status 404 returned error can't find the container with id 647b70b53df377c9dfe56ef7ae9c2233355ca3063be6785d20a65a175c4f30a4 Jul 29 07:15:21.972325 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.972297 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7c6a4b_d721_4564_9325_0c0aaf987eed.slice/crio-e59a0ed528f0b63cd2e947ec0c8ba518dd004242b5dd66729a7a27b7b12d443b WatchSource:0}: Error finding container e59a0ed528f0b63cd2e947ec0c8ba518dd004242b5dd66729a7a27b7b12d443b: Status 404 returned error can't find the container with id e59a0ed528f0b63cd2e947ec0c8ba518dd004242b5dd66729a7a27b7b12d443b Jul 29 07:15:21.972963 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.972925 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2e9088_3a0d_401b_872b_0db4b9c8315c.slice/crio-f0b4dd26c5fc2e88e2a4c36011d58516a7e40a8a35462da254a066e91131597f WatchSource:0}: Error finding container f0b4dd26c5fc2e88e2a4c36011d58516a7e40a8a35462da254a066e91131597f: Status 404 returned error can't find the container with id f0b4dd26c5fc2e88e2a4c36011d58516a7e40a8a35462da254a066e91131597f Jul 29 07:15:21.973861 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.973672 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab02368_6cde_4341_b13e_379b3edd30c6.slice/crio-e3a5810d700dfb930533a0fe1cc59a2e002cf81a7b4b8019fea55ef99e417c46 WatchSource:0}: Error finding container e3a5810d700dfb930533a0fe1cc59a2e002cf81a7b4b8019fea55ef99e417c46: Status 404 returned error can't find the container with id e3a5810d700dfb930533a0fe1cc59a2e002cf81a7b4b8019fea55ef99e417c46 Jul 29 07:15:21.975741 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.975720 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa243776_81a0_49b9_b1fa_3dc5083aaf4b.slice/crio-44f31244873fa00e0b545ab25213cfcdf3639c2ee194e6e3ab6d424ea1fd0c99 WatchSource:0}: Error finding container 44f31244873fa00e0b545ab25213cfcdf3639c2ee194e6e3ab6d424ea1fd0c99: Status 404 returned error can't find the container with id 44f31244873fa00e0b545ab25213cfcdf3639c2ee194e6e3ab6d424ea1fd0c99 Jul 29 07:15:21.976873 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.976725 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92389c6d_c9a1_471b_a501_7c646ada778e.slice/crio-91499b9772f55f46fd15bae1dfbe33476a1d5e57977d52e1370043e5dc6a2fb3 WatchSource:0}: Error finding container 91499b9772f55f46fd15bae1dfbe33476a1d5e57977d52e1370043e5dc6a2fb3: Status 404 returned error can't find the container with id 91499b9772f55f46fd15bae1dfbe33476a1d5e57977d52e1370043e5dc6a2fb3 Jul 29 07:15:21.977672 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.977620 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38d9af8_cc44_431b_b3a9_59d63f50990c.slice/crio-56c41d4a2873d9ef2384fe23d462cd750c96498b5d7a00c509dd6c6bf576a64b WatchSource:0}: Error finding container 56c41d4a2873d9ef2384fe23d462cd750c96498b5d7a00c509dd6c6bf576a64b: Status 404 returned error can't find the container with id 56c41d4a2873d9ef2384fe23d462cd750c96498b5d7a00c509dd6c6bf576a64b Jul 29 07:15:21.979646 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:15:21.979370 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb681c55a_d0f8_4e57_923a_c9f02c238ab7.slice/crio-de705eec4f7cd7e253a61e0c34c939e018180d3263f14ff87a8ae42bcd0624cf WatchSource:0}: Error finding container de705eec4f7cd7e253a61e0c34c939e018180d3263f14ff87a8ae42bcd0624cf: Status 404 returned error can't find the container with id de705eec4f7cd7e253a61e0c34c939e018180d3263f14ff87a8ae42bcd0624cf Jul 29 07:15:22.013572 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.013519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:22.013646 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.013619 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:22.013682 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.013648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:22.013725 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.013708 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:22.023147 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.023117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" event={"ID":"8ab02368-6cde-4341-b13e-379b3edd30c6","Type":"ContainerStarted","Data":"e3a5810d700dfb930533a0fe1cc59a2e002cf81a7b4b8019fea55ef99e417c46"} Jul 29 07:15:22.024862 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.024839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"f0b4dd26c5fc2e88e2a4c36011d58516a7e40a8a35462da254a066e91131597f"} Jul 29 07:15:22.026058 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.026029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r4v44" event={"ID":"9d7c6a4b-d721-4564-9325-0c0aaf987eed","Type":"ContainerStarted","Data":"e59a0ed528f0b63cd2e947ec0c8ba518dd004242b5dd66729a7a27b7b12d443b"} Jul 29 07:15:22.027153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.027132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r9jk6" event={"ID":"c362a7ff-dfdb-429b-acb5-45b263ee3950","Type":"ContainerStarted","Data":"b0828e3a689b12ba84fa6d1cb43b9962e2b777da6b1215ae3210e02d2f9b752f"} Jul 29 07:15:22.028221 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.028198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" event={"ID":"b681c55a-d0f8-4e57-923a-c9f02c238ab7","Type":"ContainerStarted","Data":"de705eec4f7cd7e253a61e0c34c939e018180d3263f14ff87a8ae42bcd0624cf"} Jul 29 07:15:22.029512 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.029489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgp6l" event={"ID":"342f3143-0c98-4b3b-b48d-a95ba0044fd7","Type":"ContainerStarted","Data":"647b70b53df377c9dfe56ef7ae9c2233355ca3063be6785d20a65a175c4f30a4"} Jul 29 07:15:22.031091 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.031060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pqrbt" event={"ID":"b38d9af8-cc44-431b-b3a9-59d63f50990c","Type":"ContainerStarted","Data":"56c41d4a2873d9ef2384fe23d462cd750c96498b5d7a00c509dd6c6bf576a64b"} Jul 29 07:15:22.032786 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.032762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerStarted","Data":"44f31244873fa00e0b545ab25213cfcdf3639c2ee194e6e3ab6d424ea1fd0c99"} Jul 29 07:15:22.034252 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.034232 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcjth" event={"ID":"92389c6d-c9a1-471b-a501-7c646ada778e","Type":"ContainerStarted","Data":"91499b9772f55f46fd15bae1dfbe33476a1d5e57977d52e1370043e5dc6a2fb3"} Jul 29 07:15:22.432855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.432568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:22.432855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.432839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:22.433085 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.432975 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:22.433085 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.433042 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:24.433023347 +0000 UTC m=+6.039201439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:22.433511 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.433418 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:22.433511 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.433464 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:15:24.43344842 +0000 UTC m=+6.039626520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:22.634456 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.634422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:22.634696 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.634604 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:22.634696 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.634622 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:22.634696 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.634635 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:22.634696 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:22.634690 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:24.634671989 +0000 UTC m=+6.240850094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:22.867399 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.867277 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 07:10:20 +0000 UTC" deadline="2027-12-28 00:48:54.731341646 +0000 UTC" Jul 29 07:15:22.867399 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:22.867344 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12401h33m31.864001605s" Jul 29 07:15:23.017251 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:23.016470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:23.017251 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:23.016610 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:23.047851 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:23.047823 2565 generic.go:356] "Generic (PLEG): container finished" podID="dfb2bb48160090119761ed3ae3f83567" containerID="b4ee13f70f4e8b28bfbb04bae6acbbe2f98bfaae1d16825fe40965a5daf3838a" exitCode=0 Jul 29 07:15:23.048002 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:23.047910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" event={"ID":"dfb2bb48160090119761ed3ae3f83567","Type":"ContainerDied","Data":"b4ee13f70f4e8b28bfbb04bae6acbbe2f98bfaae1d16825fe40965a5daf3838a"} Jul 29 07:15:23.065211 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:23.065157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" event={"ID":"e309e9e9463c00491d4bf68a40b67053","Type":"ContainerStarted","Data":"b1cc0dd415da19293a000685934a98a3b311b749f0e03928e5f6b87a1a5a3402"} Jul 29 07:15:24.013485 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.013453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:24.013932 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.013585 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:24.014042 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.014004 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:24.014278 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.014095 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:24.090867 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.090829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" event={"ID":"dfb2bb48160090119761ed3ae3f83567","Type":"ContainerStarted","Data":"ceca3c5f50bf9d910a9cca2986e80c7a8aaea2ef5127b85c9b0e0132255966bf"} Jul 29 07:15:24.105391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.105334 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-226.ec2.internal" podStartSLOduration=4.105315438 podStartE2EDuration="4.105315438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:23.080199027 +0000 UTC m=+4.686377136" watchObservedRunningTime="2026-07-29 07:15:24.105315438 +0000 UTC m=+5.711493546" Jul 29 07:15:24.449937 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.449295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:24.449937 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.449357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:24.449937 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.449474 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:24.449937 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.449506 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:24.449937 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.449544 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:15:28.449525616 +0000 UTC m=+10.055703716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:24.449937 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.449565 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:28.449554894 +0000 UTC m=+10.055732980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:24.651259 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:24.651221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:24.651446 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.651429 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:24.651517 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.651454 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:24.651517 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.651468 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:24.651628 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:24.651527 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:28.651508287 +0000 UTC m=+10.257686389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:25.014112 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:25.014082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:25.014545 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:25.014231 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:26.013582 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:26.013358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:26.013582 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:26.013478 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:26.013907 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:26.013885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:26.014619 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:26.014580 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:27.014574 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:27.014261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:27.014574 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:27.014409 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:28.013858 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:28.013824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:28.014072 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.013968 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:28.014413 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:28.014336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:28.014526 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.014435 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:28.485011 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:28.484924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:28.485011 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:28.484997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:28.485531 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.485077 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:28.485531 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.485132 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:28.485531 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.485151 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:15:36.485129612 +0000 UTC m=+18.091307703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:28.485531 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.485181 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:36.485166123 +0000 UTC m=+18.091344215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:28.686980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:28.686821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:28.687675 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.687228 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:28.687675 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.687253 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:28.687675 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.687266 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:28.687675 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:28.687329 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:36.687310307 +0000 UTC m=+18.293488395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:29.014191 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:29.014132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:29.014372 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:29.014257 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:30.013945 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:30.013911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:30.014397 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:30.014049 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:30.014397 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:30.014111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:30.014819 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:30.014229 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:31.013734 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:31.013684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:31.013917 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:31.013865 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:32.013312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:32.013279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:32.013811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:32.013279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:32.013811 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:32.013384 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:32.013811 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:32.013485 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:33.014262 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:33.014180 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:33.014684 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:33.014321 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:34.013277 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:34.013239 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:34.013467 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:34.013369 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:34.013467 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:34.013456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:34.013597 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:34.013573 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:35.014250 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:35.014215 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:35.014683 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:35.014352 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:36.013529 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:36.013497 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:36.013748 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.013625 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:36.013748 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:36.013669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:36.013918 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.013762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:36.546152 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:36.546126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:36.546509 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:36.546164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:36.546509 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.546257 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:36.546509 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.546259 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:36.546509 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.546316 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:52.546293035 +0000 UTC m=+34.152471122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:36.546509 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.546341 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:15:52.546333695 +0000 UTC m=+34.152511781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:36.747655 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:36.747618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:36.747794 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.747772 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:36.747794 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.747793 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:36.747887 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.747803 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:36.747887 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:36.747853 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:15:52.747839832 +0000 UTC m=+34.354017917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:37.013379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:37.013301 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:37.013522 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:37.013413 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:38.013393 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:38.013368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:38.013787 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:38.013483 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:38.013787 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:38.013545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:38.013787 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:38.013665 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:39.014229 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:39.014192 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:39.014624 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:39.014290 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:40.014190 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:40.014162 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:40.014190 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:40.014190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:40.014653 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:40.014272 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:40.014653 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:40.014422 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:41.014212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.013930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:41.014377 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:41.014291 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:41.128498 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.128467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pqrbt" event={"ID":"b38d9af8-cc44-431b-b3a9-59d63f50990c","Type":"ContainerStarted","Data":"4c28528eab80923b62f1ff681ae5fdd0772be2797ac5e6104608cfd3d7f2f9d8"} Jul 29 07:15:41.130056 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.130034 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa243776-81a0-49b9-b1fa-3dc5083aaf4b" containerID="eef9b8b5f28cb617713b5304cae6a8402b98cd3efdfd9e7976c5e2f770aa80b8" exitCode=0 Jul 29 07:15:41.130157 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.130107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerDied","Data":"eef9b8b5f28cb617713b5304cae6a8402b98cd3efdfd9e7976c5e2f770aa80b8"} Jul 29 07:15:41.131559 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.131532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" event={"ID":"8ab02368-6cde-4341-b13e-379b3edd30c6","Type":"ContainerStarted","Data":"4c85c5c690b28bbeebdff0befa0b11f34c5e7b3979f6c77e0e2350e86b5b81aa"} Jul 29 07:15:41.134392 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.134369 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"3d8cc1e66c8640e77280062acca363acd72917c349e35f1da7766650ed1f911a"} Jul 29 07:15:41.134477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.134437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"eedd500ad833f76840b4569888f6d040e2b66f3c37b028cc39146d0364c1bca4"} Jul 29 07:15:41.134546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.134488 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"9e3923b64c541102adad62287310a6a0289d85d19bce41f93bdd8e430f61a270"} Jul 29 07:15:41.134546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.134526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"32977b4a95b263fdb132be964b20657aec48c2b1866aee9232b54bfa19e550f2"} Jul 29 07:15:41.136123 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.135790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r4v44" event={"ID":"9d7c6a4b-d721-4564-9325-0c0aaf987eed","Type":"ContainerStarted","Data":"fe2e49b4c81419ab7f15e40ecebc958a9e8239ee0269e38ab4ef9616c9038b79"} Jul 29 07:15:41.137276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.137254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r9jk6" event={"ID":"c362a7ff-dfdb-429b-acb5-45b263ee3950","Type":"ContainerStarted","Data":"bf2c436034b25b2fda5091540e203932438d921ad6cb86620739e1d65ea94ec1"} Jul 29 07:15:41.140239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.140183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" event={"ID":"b681c55a-d0f8-4e57-923a-c9f02c238ab7","Type":"ContainerStarted","Data":"a37405ad7132530801df034c7b624ef06ca9686efb83e837a87ad087ce482663"} Jul 29 07:15:41.141806 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.141786 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sgp6l" event={"ID":"342f3143-0c98-4b3b-b48d-a95ba0044fd7","Type":"ContainerStarted","Data":"38af63a77479dea733fb9535d507c5ed5172dbe3d834c13acd14a6831a6717a6"} Jul 29 07:15:41.143503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.143469 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-226.ec2.internal" podStartSLOduration=21.14345714 podStartE2EDuration="21.14345714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:24.10639962 +0000 UTC m=+5.712577728" watchObservedRunningTime="2026-07-29 07:15:41.14345714 +0000 UTC m=+22.749635258" Jul 29 07:15:41.143757 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.143736 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pqrbt" podStartSLOduration=3.5980406670000002 podStartE2EDuration="22.143730955s" totalImagesPullingTime="18.545690288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:41.143079286 +0000 UTC m=+22.749257395" watchObservedRunningTime="2026-07-29 07:15:41.143730955 +0000 UTC m=+22.749909063" Jul 29 07:15:41.160036 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.159995 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kfrhz" podStartSLOduration=3.592535487 podStartE2EDuration="22.159981744s" totalImagesPullingTime="18.567446257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:41.159628039 +0000 UTC m=+22.765806148" watchObservedRunningTime="2026-07-29 07:15:41.159981744 +0000 UTC m=+22.766159851" Jul 29 07:15:41.175632 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.175601 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r4v44" podStartSLOduration=3.625485304 podStartE2EDuration="22.175588847s" totalImagesPullingTime="18.550103543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:41.175519457 +0000 UTC m=+22.781697565" watchObservedRunningTime="2026-07-29 07:15:41.175588847 +0000 UTC m=+22.781766955" Jul 29 07:15:41.202659 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.202622 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r9jk6" podStartSLOduration=4.5787188279999995 podStartE2EDuration="23.202610204s" totalImagesPullingTime="18.623891376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:41.202136903 +0000 UTC m=+22.808315012" watchObservedRunningTime="2026-07-29 07:15:41.202610204 +0000 UTC m=+22.808788312" Jul 29 07:15:41.219140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:41.219109 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sgp6l" podStartSLOduration=8.492428615 podStartE2EDuration="23.219097494s" totalImagesPullingTime="14.726668879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:41.218736672 +0000 UTC m=+22.824914781" watchObservedRunningTime="2026-07-29 07:15:41.219097494 +0000 UTC m=+22.825275641" Jul 29 07:15:42.013807 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.013773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:42.013807 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.013791 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:42.014058 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:42.013907 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:42.014110 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:42.014060 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:42.023161 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.023132 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 07:15:42.145425 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.145238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcjth" event={"ID":"92389c6d-c9a1-471b-a501-7c646ada778e","Type":"ContainerStarted","Data":"5dce8ddeff4f565ad3181608a6abe9cf283c374df59dce42c66eab1cc08fba30"} Jul 29 07:15:42.148880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.148852 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"aa31c122ef884053d7ec080d0f1e757d6bd0c54375e01d5480f3dbd9b49e288f"} Jul 29 07:15:42.149009 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.148924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"29f343acfdfc2ac3068e42df27e3e1d65f802eb22f964794e04eee3fcf11a521"} Jul 29 07:15:42.150781 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.150758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" event={"ID":"b681c55a-d0f8-4e57-923a-c9f02c238ab7","Type":"ContainerStarted","Data":"cd6039c710245459b2b08bf5024081f9c5cd7957e75990ba54ffbe860651be46"} Jul 29 07:15:42.172185 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.172147 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fcjth" podStartSLOduration=4.6246753290000004 podStartE2EDuration="23.172133486s" totalImagesPullingTime="18.547458157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:42.164348141 +0000 UTC m=+23.770526249" watchObservedRunningTime="2026-07-29 07:15:42.172133486 +0000 UTC m=+23.778311577" Jul 29 07:15:42.865296 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.865181 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T07:15:42.02315778Z","UUID":"8979206a-d855-44d8-8b02-7ecb780d02c0","Handler":null,"Name":"","Endpoint":""} Jul 29 07:15:42.867019 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.866994 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 07:15:42.867148 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:42.867039 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 07:15:43.013651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:43.013623 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:43.013892 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:43.013760 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:44.014223 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.014028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:44.014656 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.014089 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:44.014656 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:44.014306 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:44.014656 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:44.014388 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:44.158312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.158275 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"e1167f5e326405c285783f54882aef09b5df2cd420a7be6a6dc7c7467fa260a3"} Jul 29 07:15:44.160466 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.160438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" event={"ID":"b681c55a-d0f8-4e57-923a-c9f02c238ab7","Type":"ContainerStarted","Data":"d147d3e1a1045cc6f6b9f04d6388fc8d79e43396c1b2cbe191cffe30be0d1eae"} Jul 29 07:15:44.181327 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.181276 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hsqt" podStartSLOduration=4.391180866 podStartE2EDuration="25.181257864s" totalImagesPullingTime="20.790076998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:44.179587207 +0000 UTC m=+25.785765315" watchObservedRunningTime="2026-07-29 07:15:44.181257864 +0000 UTC m=+25.787435973" Jul 29 07:15:44.303411 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.303332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:44.304875 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:44.304694 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:45.013360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:45.013309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:45.013542 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:45.013459 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:45.163198 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:45.163164 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:45.163624 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:45.163454 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sgp6l" Jul 29 07:15:46.014094 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:46.013870 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:46.014240 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:46.013925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:46.014240 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:46.014121 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:46.014240 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:46.014223 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:46.166092 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:46.166065 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa243776-81a0-49b9-b1fa-3dc5083aaf4b" containerID="eed0da23ce2703bb082a5bd873b74dee5f77c50240b70f97d7ecf94d38bc2d07" exitCode=0 Jul 29 07:15:46.166729 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:46.166158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerDied","Data":"eed0da23ce2703bb082a5bd873b74dee5f77c50240b70f97d7ecf94d38bc2d07"} Jul 29 07:15:46.173880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:46.173856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" event={"ID":"db2e9088-3a0d-401b-872b-0db4b9c8315c","Type":"ContainerStarted","Data":"c3944e452fa0bf2cbab0a02630d85ad2536f9b043218af4dbfd924d8684fc871"} Jul 29 07:15:46.221189 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:46.221146 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" podStartSLOduration=8.610047357 podStartE2EDuration="27.221131946s" totalImagesPullingTime="18.611084589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:46.219184908 +0000 UTC m=+27.825363017" watchObservedRunningTime="2026-07-29 07:15:46.221131946 +0000 UTC m=+27.827310088" Jul 29 07:15:47.014269 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:47.014197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:47.014426 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:47.014348 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:47.175769 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:47.175743 2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 07:15:47.176231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:47.176200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:47.176344 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:47.176273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:47.190995 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:47.190976 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:47.191112 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:47.191067 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:48.013370 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:48.013336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:48.013370 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:48.013369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:48.013561 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:48.013441 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:48.013599 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:48.013554 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:48.180247 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:48.180221 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa243776-81a0-49b9-b1fa-3dc5083aaf4b" containerID="fb2aaea419cfdc99c4b563d5e9e2b906f1d5f78fefe386b61c709d6f49fbd7ad" exitCode=0 Jul 29 07:15:48.180593 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:48.180276 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerDied","Data":"fb2aaea419cfdc99c4b563d5e9e2b906f1d5f78fefe386b61c709d6f49fbd7ad"} Jul 29 07:15:48.180593 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:48.180347 2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 07:15:48.310312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:48.310255 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:15:49.015039 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:49.015005 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:49.015191 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:49.015118 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:50.013917 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:50.013882 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:50.014330 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:50.014006 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:50.014330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:50.014046 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:50.014330 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:50.014123 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:50.186671 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:50.186642 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa243776-81a0-49b9-b1fa-3dc5083aaf4b" containerID="eed92443fc0ba5d676024b9e4edb163f58d23ac332bc29a0b13e499983becf65" exitCode=0 Jul 29 07:15:50.186796 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:50.186674 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerDied","Data":"eed92443fc0ba5d676024b9e4edb163f58d23ac332bc29a0b13e499983becf65"} Jul 29 07:15:50.197655 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:50.197621 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" podUID="db2e9088-3a0d-401b-872b-0db4b9c8315c" containerName="ovnkube-controller" probeResult="failure" output="" Jul 29 07:15:51.013744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:51.013716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:51.013906 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:51.013831 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:52.013895 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:52.013870 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:52.014374 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.013981 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:52.014374 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:52.014038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:52.014374 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.014128 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:52.571243 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:52.571204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:52.571402 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:52.571264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:52.571402 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.571368 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:52.571490 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.571444 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret podName:f31a0c1a-5717-41ba-a99f-b218218bbbf6 nodeName:}" failed. No retries permitted until 2026-07-29 07:16:24.571422604 +0000 UTC m=+66.177600705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret") pod "global-pull-secret-syncer-vhxbk" (UID: "f31a0c1a-5717-41ba-a99f-b218218bbbf6") : object "kube-system"/"original-pull-secret" not registered Jul 29 07:15:52.571490 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.571368 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:52.571592 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.571531 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs podName:5e4006b4-33f1-494a-aee9-6c62a5ed904a nodeName:}" failed. No retries permitted until 2026-07-29 07:16:24.571512498 +0000 UTC m=+66.177690591 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs") pod "network-metrics-daemon-7pqft" (UID: "5e4006b4-33f1-494a-aee9-6c62a5ed904a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 07:15:52.772242 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:52.772210 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:52.772417 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.772350 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 07:15:52.772417 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.772367 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 07:15:52.772417 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.772379 2565 projected.go:196] Error preparing data for projected volume kube-api-access-rgtnx for pod openshift-network-diagnostics/network-check-target-pj9lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:52.772551 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:52.772438 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx podName:a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183 nodeName:}" failed. No retries permitted until 2026-07-29 07:16:24.772422154 +0000 UTC m=+66.378600253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rgtnx" (UniqueName: "kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx") pod "network-check-target-pj9lt" (UID: "a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 07:15:53.014321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:53.014118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:53.014321 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:53.014257 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:54.013617 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:54.013590 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:54.013617 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:54.013624 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:54.013845 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:54.013700 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:54.013845 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:54.013762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:54.196720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:54.196690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerStarted","Data":"694fbbb627feccbfc3b6951b3f3a03b52bf0d759be26e995b24c7678b3a93cfe"} Jul 29 07:15:55.013966 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:55.013937 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:55.014136 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:55.014045 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:55.200737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:55.200707 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa243776-81a0-49b9-b1fa-3dc5083aaf4b" containerID="694fbbb627feccbfc3b6951b3f3a03b52bf0d759be26e995b24c7678b3a93cfe" exitCode=0 Jul 29 07:15:55.201089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:55.200750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerDied","Data":"694fbbb627feccbfc3b6951b3f3a03b52bf0d759be26e995b24c7678b3a93cfe"} Jul 29 07:15:56.013742 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:56.013707 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:56.013893 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:56.013707 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:56.013893 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:56.013824 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:56.013893 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:56.013861 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:56.205905 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:56.205873 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa243776-81a0-49b9-b1fa-3dc5083aaf4b" containerID="04c8b374f92624b0f9081372fef1261e21bd99586ff97e0c6d3bc12feff1cb7e" exitCode=0 Jul 29 07:15:56.206227 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:56.205916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerDied","Data":"04c8b374f92624b0f9081372fef1261e21bd99586ff97e0c6d3bc12feff1cb7e"} Jul 29 07:15:57.013312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:57.013283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:57.013450 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:57.013383 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:15:57.213414 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:57.213376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" event={"ID":"aa243776-81a0-49b9-b1fa-3dc5083aaf4b","Type":"ContainerStarted","Data":"b0f52fe1717c9b162aba2ec39ad7b239d00d4f8fdb18f18e1ef62898adb9c1a4"} Jul 29 07:15:57.235895 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:57.235849 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bp8tg" podStartSLOduration=9.280584215 podStartE2EDuration="38.235836198s" totalImagesPullingTime="28.955251983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:15:57.235766023 +0000 UTC m=+38.841944132" watchObservedRunningTime="2026-07-29 07:15:57.235836198 +0000 UTC m=+38.842014303" Jul 29 07:15:58.013719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:58.013690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:15:58.013874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:58.013690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:15:58.013874 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:58.013792 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:15:58.013874 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:58.013850 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:15:59.015791 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:15:59.015765 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:15:59.016154 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:15:59.015854 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:16:00.013660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:00.013634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:00.013834 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:00.013638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:00.013834 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:00.013719 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:16:00.013834 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:00.013791 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:16:01.013945 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:01.013919 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:01.014299 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:01.014030 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:16:02.013193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:02.013171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:02.013193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:02.013199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:02.013451 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:02.013279 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:16:02.013451 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:02.013328 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:16:03.014116 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.014084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:03.014567 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:03.014230 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:16:03.579610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.579563 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pj9lt"] Jul 29 07:16:03.579807 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.579700 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:03.579887 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:03.579814 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:16:03.582066 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.582039 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vhxbk"] Jul 29 07:16:03.582180 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.582148 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:03.582273 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:03.582251 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:16:03.593592 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.593569 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7pqft"] Jul 29 07:16:03.593690 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:03.593643 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:03.593735 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:03.593721 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:16:05.013610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:05.013576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:05.014317 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:05.013699 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:16:06.013574 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:06.013546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:06.013721 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:06.013635 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhxbk" podUID="f31a0c1a-5717-41ba-a99f-b218218bbbf6" Jul 29 07:16:06.013721 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:06.013690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:06.014122 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:06.013782 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pqft" podUID="5e4006b4-33f1-494a-aee9-6c62a5ed904a" Jul 29 07:16:07.014055 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.014026 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:07.014499 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:07.014148 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pj9lt" podUID="a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183" Jul 29 07:16:07.183700 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.183605 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-226.ec2.internal" event="NodeReady" Jul 29 07:16:07.183818 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.183791 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 07:16:07.231389 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.231360 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fd6c64cd-c89jh"] Jul 29 07:16:07.252200 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.252178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.253343 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.253324 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4djsx"] Jul 29 07:16:07.256195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.256166 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 07:16:07.256195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.256191 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-846cg\"" Jul 29 07:16:07.256366 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.256349 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 07:16:07.256476 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.256461 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 07:16:07.265784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.265715 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 07:16:07.265784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.265766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.268559 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.268537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd6c64cd-c89jh"] Jul 29 07:16:07.269133 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.269115 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qvxl2"] Jul 29 07:16:07.270787 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.270768 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6q689\"" Jul 29 07:16:07.271528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.271511 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 07:16:07.271631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.271510 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 07:16:07.272753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbea23ef-06fe-448e-b3da-793ce401fac1-image-registry-private-configuration\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.272835 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzh4\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-kube-api-access-zqzh4\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.272835 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbea23ef-06fe-448e-b3da-793ce401fac1-registry-certificates\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.272971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8cb8b35-e0d0-46cf-9d36-ab496045c246-metrics-tls\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.272971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mh9\" (UniqueName: \"kubernetes.io/projected/d8cb8b35-e0d0-46cf-9d36-ab496045c246-kube-api-access-28mh9\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.272971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-registry-tls\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.273127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-bound-sa-token\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.273127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.272997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8cb8b35-e0d0-46cf-9d36-ab496045c246-config-volume\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.273127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.273031 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbea23ef-06fe-448e-b3da-793ce401fac1-ca-trust-extracted\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.273127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.273054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbea23ef-06fe-448e-b3da-793ce401fac1-trusted-ca\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.273127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.273075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8cb8b35-e0d0-46cf-9d36-ab496045c246-tmp-dir\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.273127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.273101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbea23ef-06fe-448e-b3da-793ce401fac1-installation-pull-secrets\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.292069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.292050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4djsx"] Jul 29 07:16:07.292149 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.292081 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qvxl2"] Jul 29 07:16:07.292187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.292149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.295538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.295516 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zj4vm\"" Jul 29 07:16:07.295891 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.295878 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 07:16:07.296187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.296171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 07:16:07.296379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.296367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 07:16:07.296463 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.296448 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 07:16:07.346155 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.346136 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-55xw5"] Jul 29 07:16:07.359538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.359518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55xw5"] Jul 29 07:16:07.359626 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.359612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.362283 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.362267 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 07:16:07.362379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.362340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6scvq\"" Jul 29 07:16:07.362480 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.362382 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 07:16:07.362480 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.362475 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 07:16:07.374177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbea23ef-06fe-448e-b3da-793ce401fac1-installation-pull-secrets\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0faf0f45-8518-4a64-9506-e499f0d4e3fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.374264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h995m\" (UniqueName: \"kubernetes.io/projected/0faf0f45-8518-4a64-9506-e499f0d4e3fa-kube-api-access-h995m\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.374264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbea23ef-06fe-448e-b3da-793ce401fac1-image-registry-private-configuration\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzh4\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-kube-api-access-zqzh4\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97771b42-fccd-49b0-9b43-f7d1869ace3a-cert\") pod \"ingress-canary-55xw5\" (UID: \"97771b42-fccd-49b0-9b43-f7d1869ace3a\") " pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.374398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbea23ef-06fe-448e-b3da-793ce401fac1-registry-certificates\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8cb8b35-e0d0-46cf-9d36-ab496045c246-metrics-tls\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.374601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mh9\" (UniqueName: \"kubernetes.io/projected/d8cb8b35-e0d0-46cf-9d36-ab496045c246-kube-api-access-28mh9\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.374601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faf0f45-8518-4a64-9506-e499f0d4e3fa-data-volume\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.374601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-registry-tls\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6xx\" (UniqueName: \"kubernetes.io/projected/97771b42-fccd-49b0-9b43-f7d1869ace3a-kube-api-access-7l6xx\") pod \"ingress-canary-55xw5\" (UID: \"97771b42-fccd-49b0-9b43-f7d1869ace3a\") " pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.374601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-bound-sa-token\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8cb8b35-e0d0-46cf-9d36-ab496045c246-config-volume\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.374919 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbea23ef-06fe-448e-b3da-793ce401fac1-ca-trust-extracted\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374919 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbea23ef-06fe-448e-b3da-793ce401fac1-trusted-ca\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.374919 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8cb8b35-e0d0-46cf-9d36-ab496045c246-tmp-dir\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.374919 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0faf0f45-8518-4a64-9506-e499f0d4e3fa-crio-socket\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.374919 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.374704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0faf0f45-8518-4a64-9506-e499f0d4e3fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.375145 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.375040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbea23ef-06fe-448e-b3da-793ce401fac1-ca-trust-extracted\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.375145 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.375107 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbea23ef-06fe-448e-b3da-793ce401fac1-registry-certificates\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.375232 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.375186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d8cb8b35-e0d0-46cf-9d36-ab496045c246-config-volume\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.375548 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.375374 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8cb8b35-e0d0-46cf-9d36-ab496045c246-tmp-dir\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.376446 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.376421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbea23ef-06fe-448e-b3da-793ce401fac1-trusted-ca\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.378336 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.378316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbea23ef-06fe-448e-b3da-793ce401fac1-image-registry-private-configuration\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.378433 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.378353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8cb8b35-e0d0-46cf-9d36-ab496045c246-metrics-tls\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.378493 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.378436 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-registry-tls\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.378493 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.378460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbea23ef-06fe-448e-b3da-793ce401fac1-installation-pull-secrets\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.384921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.384893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mh9\" (UniqueName: \"kubernetes.io/projected/d8cb8b35-e0d0-46cf-9d36-ab496045c246-kube-api-access-28mh9\") pod \"dns-default-4djsx\" (UID: \"d8cb8b35-e0d0-46cf-9d36-ab496045c246\") " pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.385474 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.385450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-bound-sa-token\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.385742 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.385725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzh4\" (UniqueName: \"kubernetes.io/projected/cbea23ef-06fe-448e-b3da-793ce401fac1-kube-api-access-zqzh4\") pod \"image-registry-7fd6c64cd-c89jh\" (UID: \"cbea23ef-06fe-448e-b3da-793ce401fac1\") " pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.475765 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.475740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faf0f45-8518-4a64-9506-e499f0d4e3fa-data-volume\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.475871 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.475847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6xx\" (UniqueName: \"kubernetes.io/projected/97771b42-fccd-49b0-9b43-f7d1869ace3a-kube-api-access-7l6xx\") pod \"ingress-canary-55xw5\" (UID: \"97771b42-fccd-49b0-9b43-f7d1869ace3a\") " pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.475934 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.475887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0faf0f45-8518-4a64-9506-e499f0d4e3fa-crio-socket\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.475934 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.475903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0faf0f45-8518-4a64-9506-e499f0d4e3fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.476070 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.475933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0faf0f45-8518-4a64-9506-e499f0d4e3fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.476070 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.475971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h995m\" (UniqueName: \"kubernetes.io/projected/0faf0f45-8518-4a64-9506-e499f0d4e3fa-kube-api-access-h995m\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.476070 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.476001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97771b42-fccd-49b0-9b43-f7d1869ace3a-cert\") pod \"ingress-canary-55xw5\" (UID: \"97771b42-fccd-49b0-9b43-f7d1869ace3a\") " pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.476218 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.476073 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faf0f45-8518-4a64-9506-e499f0d4e3fa-data-volume\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.476218 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.476093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0faf0f45-8518-4a64-9506-e499f0d4e3fa-crio-socket\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.476415 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.476397 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0faf0f45-8518-4a64-9506-e499f0d4e3fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.478008 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.477965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0faf0f45-8518-4a64-9506-e499f0d4e3fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.478111 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.478096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97771b42-fccd-49b0-9b43-f7d1869ace3a-cert\") pod \"ingress-canary-55xw5\" (UID: \"97771b42-fccd-49b0-9b43-f7d1869ace3a\") " pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.487722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.487697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6xx\" (UniqueName: \"kubernetes.io/projected/97771b42-fccd-49b0-9b43-f7d1869ace3a-kube-api-access-7l6xx\") pod \"ingress-canary-55xw5\" (UID: \"97771b42-fccd-49b0-9b43-f7d1869ace3a\") " pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.488118 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.488098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h995m\" (UniqueName: \"kubernetes.io/projected/0faf0f45-8518-4a64-9506-e499f0d4e3fa-kube-api-access-h995m\") pod \"insights-runtime-extractor-qvxl2\" (UID: \"0faf0f45-8518-4a64-9506-e499f0d4e3fa\") " pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.562071 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.562049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:07.574336 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.574317 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:07.600100 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.600083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvxl2" Jul 29 07:16:07.667232 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.667207 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55xw5" Jul 29 07:16:07.813871 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.813790 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4djsx"] Jul 29 07:16:07.825714 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.825692 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qvxl2"] Jul 29 07:16:07.827770 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:07.827747 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cb8b35_e0d0_46cf_9d36_ab496045c246.slice/crio-6b0dc111c6fead8d597d18185584b8461b3df701d0bb7e257f4a2463233555e6 WatchSource:0}: Error finding container 6b0dc111c6fead8d597d18185584b8461b3df701d0bb7e257f4a2463233555e6: Status 404 returned error can't find the container with id 6b0dc111c6fead8d597d18185584b8461b3df701d0bb7e257f4a2463233555e6 Jul 29 07:16:07.828257 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:07.828239 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0faf0f45_8518_4a64_9506_e499f0d4e3fa.slice/crio-85341483092c9c728f2f224fa2413f88f6f36f9754f1960920623f4ceb80ebfa WatchSource:0}: Error finding container 85341483092c9c728f2f224fa2413f88f6f36f9754f1960920623f4ceb80ebfa: Status 404 returned error can't find the container with id 85341483092c9c728f2f224fa2413f88f6f36f9754f1960920623f4ceb80ebfa Jul 29 07:16:07.831570 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.831545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd6c64cd-c89jh"] Jul 29 07:16:07.833619 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:07.833598 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea23ef_06fe_448e_b3da_793ce401fac1.slice/crio-0cb6a34b8683de01d1703b746fc8a4d2d806a60e9b0aa4dfa45b05e13caf68be WatchSource:0}: Error finding container 0cb6a34b8683de01d1703b746fc8a4d2d806a60e9b0aa4dfa45b05e13caf68be: Status 404 returned error can't find the container with id 0cb6a34b8683de01d1703b746fc8a4d2d806a60e9b0aa4dfa45b05e13caf68be Jul 29 07:16:07.864658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:07.864639 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55xw5"] Jul 29 07:16:07.879006 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:07.878979 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97771b42_fccd_49b0_9b43_f7d1869ace3a.slice/crio-d39aebe0ea04b9084a4ddc59a3204335e252274aa9575cd5519e7511460d66ad WatchSource:0}: Error finding container d39aebe0ea04b9084a4ddc59a3204335e252274aa9575cd5519e7511460d66ad: Status 404 returned error can't find the container with id d39aebe0ea04b9084a4ddc59a3204335e252274aa9575cd5519e7511460d66ad Jul 29 07:16:08.013544 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.013391 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:08.013668 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.013440 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:08.016603 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.016578 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 07:16:08.016603 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.016596 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 07:16:08.017063 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.016701 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb4zn\"" Jul 29 07:16:08.242842 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.242762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4djsx" event={"ID":"d8cb8b35-e0d0-46cf-9d36-ab496045c246","Type":"ContainerStarted","Data":"6b0dc111c6fead8d597d18185584b8461b3df701d0bb7e257f4a2463233555e6"} Jul 29 07:16:08.244210 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.244181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" event={"ID":"cbea23ef-06fe-448e-b3da-793ce401fac1","Type":"ContainerStarted","Data":"82d83ddbb42ea0de33ea19a8183061a89d64c1034d87ea3a1d710af28d3cc92f"} Jul 29 07:16:08.244342 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.244234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" event={"ID":"cbea23ef-06fe-448e-b3da-793ce401fac1","Type":"ContainerStarted","Data":"0cb6a34b8683de01d1703b746fc8a4d2d806a60e9b0aa4dfa45b05e13caf68be"} Jul 29 07:16:08.244342 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.244264 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:08.245306 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.245280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvxl2" event={"ID":"0faf0f45-8518-4a64-9506-e499f0d4e3fa","Type":"ContainerStarted","Data":"85341483092c9c728f2f224fa2413f88f6f36f9754f1960920623f4ceb80ebfa"} Jul 29 07:16:08.246436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.246408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55xw5" event={"ID":"97771b42-fccd-49b0-9b43-f7d1869ace3a","Type":"ContainerStarted","Data":"d39aebe0ea04b9084a4ddc59a3204335e252274aa9575cd5519e7511460d66ad"} Jul 29 07:16:08.271637 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:08.271564 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" podStartSLOduration=6.271538332 podStartE2EDuration="6.271538332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:08.269152535 +0000 UTC m=+49.875330644" watchObservedRunningTime="2026-07-29 07:16:08.271538332 +0000 UTC m=+49.877716475" Jul 29 07:16:09.018824 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:09.017746 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:09.023639 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:09.021739 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 07:16:09.023639 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:09.022089 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rkrh4\"" Jul 29 07:16:09.023999 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:09.023932 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 07:16:14.263430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:14.263184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55xw5" event={"ID":"97771b42-fccd-49b0-9b43-f7d1869ace3a","Type":"ContainerStarted","Data":"a7315e18720d205da8318be684b36eee9983c44413daed1c919ca769c8aa3c87"} Jul 29 07:16:14.265440 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:14.265416 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4djsx" event={"ID":"d8cb8b35-e0d0-46cf-9d36-ab496045c246","Type":"ContainerStarted","Data":"49e8aaf4bbabc5e9afe5d4d8058e432760147537befb400bc7a25604a6a0a12d"} Jul 29 07:16:14.265563 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:14.265459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4djsx" event={"ID":"d8cb8b35-e0d0-46cf-9d36-ab496045c246","Type":"ContainerStarted","Data":"02aa4f98ec8ad5cfce3abd4026dabf854929469ea59d3c4dce9c5065767d1f4f"} Jul 29 07:16:14.265620 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:14.265572 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:14.280071 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:14.279895 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-55xw5" podStartSLOduration=1.175885165 podStartE2EDuration="7.279875835s" totalImagesPullingTime="6.10399067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:14.2791115 +0000 UTC m=+55.885289609" watchObservedRunningTime="2026-07-29 07:16:14.279875835 +0000 UTC m=+55.886053944" Jul 29 07:16:14.300085 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:14.300036 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4djsx" podStartSLOduration=1.1450779500000001 podStartE2EDuration="7.300015616s" totalImagesPullingTime="6.154937666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:14.299581084 +0000 UTC m=+55.905759192" watchObservedRunningTime="2026-07-29 07:16:14.300015616 +0000 UTC m=+55.906193724" Jul 29 07:16:15.271017 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:15.270977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvxl2" event={"ID":"0faf0f45-8518-4a64-9506-e499f0d4e3fa","Type":"ContainerStarted","Data":"dcdc4e1988303397d000065bb5f78cecfd22ac201c0e4d49372e9f07b24ca74c"} Jul 29 07:16:18.280419 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:18.280341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvxl2" event={"ID":"0faf0f45-8518-4a64-9506-e499f0d4e3fa","Type":"ContainerStarted","Data":"64d0dc917492099369f169dbac090c866b432f7b75de54fed86ba78edf1c8fac"} Jul 29 07:16:20.198659 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:20.198625 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8cg7x" Jul 29 07:16:22.293819 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:22.293784 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvxl2" event={"ID":"0faf0f45-8518-4a64-9506-e499f0d4e3fa","Type":"ContainerStarted","Data":"81665e490537aaf9b08a08e43bb1f8cdc7473d8a67124e2e4411acf16378cb5f"} Jul 29 07:16:22.312883 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:22.312842 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qvxl2" podStartSLOduration=1.943081532 podStartE2EDuration="15.312827193s" totalImagesPullingTime="13.369745661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:16:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:22.312598733 +0000 UTC m=+63.918776842" watchObservedRunningTime="2026-07-29 07:16:22.312827193 +0000 UTC m=+63.919005300" Jul 29 07:16:24.217427 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.217393 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd"] Jul 29 07:16:24.220039 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.220024 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.223068 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.223045 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 07:16:24.223528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.223511 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 07:16:24.225099 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.225079 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 07:16:24.225099 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.225089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 07:16:24.225296 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.225099 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 07:16:24.225296 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.225139 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lp9k9\"" Jul 29 07:16:24.225296 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.225273 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 07:16:24.235791 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.235768 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd"] Jul 29 07:16:24.243658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.243639 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5br72"] Jul 29 07:16:24.245552 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.245538 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.248312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.248296 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 07:16:24.248562 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.248543 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ph7j\"" Jul 29 07:16:24.248562 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.248557 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 07:16:24.248694 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.248562 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 07:16:24.272845 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.272829 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4djsx" Jul 29 07:16:24.308081 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-root\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308180 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmj6k\" (UniqueName: \"kubernetes.io/projected/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-api-access-bmj6k\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.308180 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-sys\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308180 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.308295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.308295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x999c\" (UniqueName: \"kubernetes.io/projected/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-kube-api-access-x999c\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-metrics-client-ca\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.308530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-textfile\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-wtmp\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-accelerators-collector-config\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.308700 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.308700 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.308559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.409162 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmj6k\" (UniqueName: \"kubernetes.io/projected/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-api-access-bmj6k\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.409315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-sys\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.409315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.409315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-sys\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409576 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x999c\" (UniqueName: \"kubernetes.io/projected/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-kube-api-access-x999c\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409576 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-metrics-client-ca\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409576 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409576 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409569 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.409761 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.409761 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-textfile\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409761 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409761 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-wtmp\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-wtmp\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-textfile\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.409924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-accelerators-collector-config\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.409980 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:24.409940 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 07:16:24.410142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.410142 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:24.410028 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls podName:7177cd99-fdf1-49b1-9ca2-70739ef9c4de nodeName:}" failed. No retries permitted until 2026-07-29 07:16:24.910010189 +0000 UTC m=+66.516188281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls") pod "node-exporter-5br72" (UID: "7177cd99-fdf1-49b1-9ca2-70739ef9c4de") : secret "node-exporter-tls" not found Jul 29 07:16:24.410142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.410142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.410142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-root\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.410348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-root\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.410348 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:24.410227 2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 29 07:16:24.410348 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:24.410264 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-tls podName:459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0 nodeName:}" failed. No retries permitted until 2026-07-29 07:16:24.910253655 +0000 UTC m=+66.516431741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-stcxd" (UID: "459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0") : secret "kube-state-metrics-tls" not found Jul 29 07:16:24.410477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-accelerators-collector-config\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.410477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-metrics-client-ca\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.410603 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.410581 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.413082 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.413064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.413173 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.413144 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.418839 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.418816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x999c\" (UniqueName: \"kubernetes.io/projected/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-kube-api-access-x999c\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.420241 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.420219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmj6k\" (UniqueName: \"kubernetes.io/projected/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-api-access-bmj6k\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.611658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.611623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:24.611797 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.611671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:24.614457 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.614439 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 07:16:24.614637 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.614624 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 07:16:24.623838 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.623817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31a0c1a-5717-41ba-a99f-b218218bbbf6-original-pull-secret\") pod \"global-pull-secret-syncer-vhxbk\" (UID: \"f31a0c1a-5717-41ba-a99f-b218218bbbf6\") " pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:24.623988 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.623968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4006b4-33f1-494a-aee9-6c62a5ed904a-metrics-certs\") pod \"network-metrics-daemon-7pqft\" (UID: \"5e4006b4-33f1-494a-aee9-6c62a5ed904a\") " pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:24.813565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.813534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:24.816439 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.816423 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 07:16:24.825453 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.825430 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb4zn\"" Jul 29 07:16:24.826441 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.826428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 07:16:24.827518 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.827504 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhxbk" Jul 29 07:16:24.833911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.833888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pqft" Jul 29 07:16:24.836301 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.836280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtnx\" (UniqueName: \"kubernetes.io/projected/a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183-kube-api-access-rgtnx\") pod \"network-check-target-pj9lt\" (UID: \"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183\") " pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:24.914518 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.914490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.914631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.914557 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:24.914702 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:24.914653 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 07:16:24.914702 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:24.914700 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls podName:7177cd99-fdf1-49b1-9ca2-70739ef9c4de nodeName:}" failed. No retries permitted until 2026-07-29 07:16:25.91468842 +0000 UTC m=+67.520866505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls") pod "node-exporter-5br72" (UID: "7177cd99-fdf1-49b1-9ca2-70739ef9c4de") : secret "node-exporter-tls" not found Jul 29 07:16:24.916715 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.916689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-stcxd\" (UID: \"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:24.936317 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.936290 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rkrh4\"" Jul 29 07:16:24.943660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.943638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:24.954588 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.954567 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vhxbk"] Jul 29 07:16:24.956542 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:24.956512 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31a0c1a_5717_41ba_a99f_b218218bbbf6.slice/crio-76cfaf5f97fad6729623b2666b927a438c689f3bfaa96f31cbb181307ed66f6e WatchSource:0}: Error finding container 76cfaf5f97fad6729623b2666b927a438c689f3bfaa96f31cbb181307ed66f6e: Status 404 returned error can't find the container with id 76cfaf5f97fad6729623b2666b927a438c689f3bfaa96f31cbb181307ed66f6e Jul 29 07:16:24.973267 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:24.973204 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7pqft"] Jul 29 07:16:24.975352 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:24.975316 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4006b4_33f1_494a_aee9_6c62a5ed904a.slice/crio-94b79588c352f3ddbbcbf214facee98384ab1e7433851370469b9d84044cbc10 WatchSource:0}: Error finding container 94b79588c352f3ddbbcbf214facee98384ab1e7433851370469b9d84044cbc10: Status 404 returned error can't find the container with id 94b79588c352f3ddbbcbf214facee98384ab1e7433851370469b9d84044cbc10 Jul 29 07:16:25.077131 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.077102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pj9lt"] Jul 29 07:16:25.079732 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:25.079706 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bf4d8a_4663_4d6e_8ed5_5bb549ffc183.slice/crio-c46e7a92e2e7f0f616718fdc46b19d43fe73b3a88f7f458b3e92b6595a25ee06 WatchSource:0}: Error finding container c46e7a92e2e7f0f616718fdc46b19d43fe73b3a88f7f458b3e92b6595a25ee06: Status 404 returned error can't find the container with id c46e7a92e2e7f0f616718fdc46b19d43fe73b3a88f7f458b3e92b6595a25ee06 Jul 29 07:16:25.128520 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.128472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" Jul 29 07:16:25.248082 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.248060 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd"] Jul 29 07:16:25.250333 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:25.250307 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459fc0f5_ba7b_4b9c_aa6a_4687ea13f8e0.slice/crio-693f58834b2b8ad0fa2f73742cecfe4c821ab72c1e0bdbb189fc157e107dae1f WatchSource:0}: Error finding container 693f58834b2b8ad0fa2f73742cecfe4c821ab72c1e0bdbb189fc157e107dae1f: Status 404 returned error can't find the container with id 693f58834b2b8ad0fa2f73742cecfe4c821ab72c1e0bdbb189fc157e107dae1f Jul 29 07:16:25.305806 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.305527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7pqft" event={"ID":"5e4006b4-33f1-494a-aee9-6c62a5ed904a","Type":"ContainerStarted","Data":"94b79588c352f3ddbbcbf214facee98384ab1e7433851370469b9d84044cbc10"} Jul 29 07:16:25.307182 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.307153 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vhxbk" event={"ID":"f31a0c1a-5717-41ba-a99f-b218218bbbf6","Type":"ContainerStarted","Data":"76cfaf5f97fad6729623b2666b927a438c689f3bfaa96f31cbb181307ed66f6e"} Jul 29 07:16:25.308186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.308159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pj9lt" event={"ID":"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183","Type":"ContainerStarted","Data":"c46e7a92e2e7f0f616718fdc46b19d43fe73b3a88f7f458b3e92b6595a25ee06"} Jul 29 07:16:25.309130 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.309112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" event={"ID":"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0","Type":"ContainerStarted","Data":"693f58834b2b8ad0fa2f73742cecfe4c821ab72c1e0bdbb189fc157e107dae1f"} Jul 29 07:16:25.324572 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.324552 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:16:25.327375 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.327360 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.330769 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.330748 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 07:16:25.331069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331046 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 07:16:25.331069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331059 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 07:16:25.331231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331052 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 07:16:25.331231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331103 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ddxxn\"" Jul 29 07:16:25.331231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331059 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 07:16:25.331231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331053 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 07:16:25.331426 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 07:16:25.331426 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331269 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 07:16:25.331426 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.331336 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 07:16:25.348456 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.348428 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:16:25.417773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.417773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.417773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75s8\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-kube-api-access-q75s8\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-web-config\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.417942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-out\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.418000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.418035 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.418065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.418079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.418330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.418131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.518860 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.518843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.518972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.518874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.518972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.518893 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.518972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.518914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.518972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.518941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75s8\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-kube-api-access-q75s8\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519223 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519223 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-web-config\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519223 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519223 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-out\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519223 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519473 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:16:25.519287 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle podName:d7c6a7fe-29c4-4891-8fd2-11f11d767b69 nodeName:}" failed. No retries permitted until 2026-07-29 07:16:26.019265285 +0000 UTC m=+67.625443389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69") : configmap references non-existent config key: ca-bundle.crt Jul 29 07:16:25.519473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.519473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.519368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.520396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.520367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.521525 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.521478 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-out\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.522372 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.522046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.522372 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.522368 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.522817 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.522596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.523084 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.523058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.523488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.523466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.523729 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.523713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.523941 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.523902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-web-config\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.524491 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.524468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.527907 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.527880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75s8\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-kube-api-access-q75s8\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:25.922354 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.922316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:25.925166 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:25.925134 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7177cd99-fdf1-49b1-9ca2-70739ef9c4de-node-exporter-tls\") pod \"node-exporter-5br72\" (UID: \"7177cd99-fdf1-49b1-9ca2-70739ef9c4de\") " pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:26.022876 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:26.022842 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:26.023805 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:26.023761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:26.053741 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:26.053716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5br72" Jul 29 07:16:26.070168 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:26.070142 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7177cd99_fdf1_49b1_9ca2_70739ef9c4de.slice/crio-d65a8b45963b0da2bd889c9b4b1c1b3ea2c9945d8be071834e5beb8c5ac73cf2 WatchSource:0}: Error finding container d65a8b45963b0da2bd889c9b4b1c1b3ea2c9945d8be071834e5beb8c5ac73cf2: Status 404 returned error can't find the container with id d65a8b45963b0da2bd889c9b4b1c1b3ea2c9945d8be071834e5beb8c5ac73cf2 Jul 29 07:16:26.236498 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:26.236423 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:16:26.315411 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:26.315377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5br72" event={"ID":"7177cd99-fdf1-49b1-9ca2-70739ef9c4de","Type":"ContainerStarted","Data":"d65a8b45963b0da2bd889c9b4b1c1b3ea2c9945d8be071834e5beb8c5ac73cf2"} Jul 29 07:16:27.052625 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.052379 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:16:27.053417 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:27.053387 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c6a7fe_29c4_4891_8fd2_11f11d767b69.slice/crio-ce74c56762ad55a3684f402e5e4786d277955bcfb21a0e29040c95fb4d8d4d28 WatchSource:0}: Error finding container ce74c56762ad55a3684f402e5e4786d277955bcfb21a0e29040c95fb4d8d4d28: Status 404 returned error can't find the container with id ce74c56762ad55a3684f402e5e4786d277955bcfb21a0e29040c95fb4d8d4d28 Jul 29 07:16:27.303116 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.302364 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6898c4698c-gl8g2"] Jul 29 07:16:27.322927 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.321863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.322927 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.322810 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6898c4698c-gl8g2"] Jul 29 07:16:27.325851 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.324860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"ce74c56762ad55a3684f402e5e4786d277955bcfb21a0e29040c95fb4d8d4d28"} Jul 29 07:16:27.326387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.326368 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 29 07:16:27.326783 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.326765 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lhdlp\"" Jul 29 07:16:27.327060 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.327044 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ckug03uffoer8\"" Jul 29 07:16:27.327664 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.327270 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 29 07:16:27.327664 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.327399 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 29 07:16:27.327823 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.327687 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 29 07:16:27.327823 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.327694 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 29 07:16:27.333719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.329380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" event={"ID":"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0","Type":"ContainerStarted","Data":"100a2e9a7faf27edc2a9486806b7209cf5291f4a1dafb70e0c715ac28d148960"} Jul 29 07:16:27.333719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.329431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" event={"ID":"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0","Type":"ContainerStarted","Data":"f4ac72da27038295ba171f467416496f6f8731f81e1a20add68e04f91415fd7e"} Jul 29 07:16:27.337414 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.337385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7pqft" event={"ID":"5e4006b4-33f1-494a-aee9-6c62a5ed904a","Type":"ContainerStarted","Data":"dc2511bab48ecec808ab44dc3e3dd26deaedfd1f1827b5ce0be7840e04d2bc14"} Jul 29 07:16:27.337524 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.337431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7pqft" event={"ID":"5e4006b4-33f1-494a-aee9-6c62a5ed904a","Type":"ContainerStarted","Data":"46455432011dc5cb3f86c8ce17bd84109ee31fd12e48810787399c60e7c6b0f4"} Jul 29 07:16:27.375160 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.375107 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7pqft" podStartSLOduration=66.431373621 podStartE2EDuration="1m8.375085973s" totalImagesPullingTime="1.943712352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:27.373196725 +0000 UTC m=+68.979374841" watchObservedRunningTime="2026-07-29 07:16:27.375085973 +0000 UTC m=+68.981264081" Jul 29 07:16:27.437609 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.437609 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.437803 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19e3cb62-307a-40af-b497-3afb406dc032-metrics-client-ca\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.437803 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.437803 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-tls\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.437803 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-grpc-tls\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.438025 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5265v\" (UniqueName: \"kubernetes.io/projected/19e3cb62-307a-40af-b497-3afb406dc032-kube-api-access-5265v\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.438025 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.437863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538501 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19e3cb62-307a-40af-b497-3afb406dc032-metrics-client-ca\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-tls\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538937 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-grpc-tls\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.538937 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.538733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5265v\" (UniqueName: \"kubernetes.io/projected/19e3cb62-307a-40af-b497-3afb406dc032-kube-api-access-5265v\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.540781 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.540731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19e3cb62-307a-40af-b497-3afb406dc032-metrics-client-ca\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.542786 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.542738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.544095 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.543771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.544095 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.543863 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.544095 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.544055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-tls\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.544286 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.544130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.545107 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.545068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/19e3cb62-307a-40af-b497-3afb406dc032-secret-grpc-tls\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.549871 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.549831 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5265v\" (UniqueName: \"kubernetes.io/projected/19e3cb62-307a-40af-b497-3afb406dc032-kube-api-access-5265v\") pod \"thanos-querier-6898c4698c-gl8g2\" (UID: \"19e3cb62-307a-40af-b497-3afb406dc032\") " pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:27.568702 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.568662 2565 patch_prober.go:28] interesting pod/image-registry-7fd6c64cd-c89jh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 07:16:27.568799 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.568724 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" podUID="cbea23ef-06fe-448e-b3da-793ce401fac1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:16:27.637128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:27.636871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:28.136307 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:28.136278 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6898c4698c-gl8g2"] Jul 29 07:16:28.343492 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:28.343452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" event={"ID":"459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0","Type":"ContainerStarted","Data":"bd049c87d10d2e1187c6a3db5044813accc1f575d6d88b056a45a75a680ad723"} Jul 29 07:16:28.366558 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:28.366509 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-stcxd" podStartSLOduration=2.693451498 podStartE2EDuration="4.366493332s" totalImagesPullingTime="1.673041834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:28.364479815 +0000 UTC m=+69.970657934" watchObservedRunningTime="2026-07-29 07:16:28.366493332 +0000 UTC m=+69.972671438" Jul 29 07:16:28.511560 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:28.511528 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e3cb62_307a_40af_b497_3afb406dc032.slice/crio-bf4232fc7cdc0d759898883efccc3b9fd3d8c8a4ac996e3196b93b8059f7e4cc WatchSource:0}: Error finding container bf4232fc7cdc0d759898883efccc3b9fd3d8c8a4ac996e3196b93b8059f7e4cc: Status 404 returned error can't find the container with id bf4232fc7cdc0d759898883efccc3b9fd3d8c8a4ac996e3196b93b8059f7e4cc Jul 29 07:16:29.253724 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.253684 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fd6c64cd-c89jh" Jul 29 07:16:29.348481 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.348446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"bf4232fc7cdc0d759898883efccc3b9fd3d8c8a4ac996e3196b93b8059f7e4cc"} Jul 29 07:16:29.444171 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.444139 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7"] Jul 29 07:16:29.447096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.447074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.451286 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.451264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-96l68\"" Jul 29 07:16:29.451716 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.451698 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 07:16:29.451839 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.451745 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 07:16:29.451895 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.451847 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 07:16:29.452117 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.452093 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 07:16:29.452222 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.452127 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 07:16:29.460261 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.460239 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 07:16:29.463091 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.463066 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7"] Jul 29 07:16:29.556122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-metrics-client-ca\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556322 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-serving-certs-ca-bundle\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556322 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nknt\" (UniqueName: \"kubernetes.io/projected/b2453c3d-97fb-45f2-8062-cb27f05b8717-kube-api-access-9nknt\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556322 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-secret-telemeter-client\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556435 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556435 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-federate-client-tls\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.556524 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.556437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-telemeter-client-tls\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657397 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-federate-client-tls\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-telemeter-client-tls\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-metrics-client-ca\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-serving-certs-ca-bundle\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nknt\" (UniqueName: \"kubernetes.io/projected/b2453c3d-97fb-45f2-8062-cb27f05b8717-kube-api-access-9nknt\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.657852 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.657587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-secret-telemeter-client\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.659244 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.658619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-metrics-client-ca\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.659244 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.658756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.659244 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.659198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2453c3d-97fb-45f2-8062-cb27f05b8717-serving-certs-ca-bundle\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.660566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.660541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-secret-telemeter-client\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.660801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.660779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-federate-client-tls\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.660846 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.660819 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-telemeter-client-tls\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.660922 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.660842 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2453c3d-97fb-45f2-8062-cb27f05b8717-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.666401 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.666381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nknt\" (UniqueName: \"kubernetes.io/projected/b2453c3d-97fb-45f2-8062-cb27f05b8717-kube-api-access-9nknt\") pod \"telemeter-client-56d4bf98f7-fw9z7\" (UID: \"b2453c3d-97fb-45f2-8062-cb27f05b8717\") " pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:29.758888 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:29.758861 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" Jul 29 07:16:31.060749 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.060661 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7"] Jul 29 07:16:31.061995 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:31.061918 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2453c3d_97fb_45f2_8062_cb27f05b8717.slice/crio-abc2147ac9e7611b31cd6e6565918dffde26467a3996ec7a1df7850e7ed6b13d WatchSource:0}: Error finding container abc2147ac9e7611b31cd6e6565918dffde26467a3996ec7a1df7850e7ed6b13d: Status 404 returned error can't find the container with id abc2147ac9e7611b31cd6e6565918dffde26467a3996ec7a1df7850e7ed6b13d Jul 29 07:16:31.358072 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.358038 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pj9lt" event={"ID":"a4bf4d8a-4663-4d6e-8ed5-5bb549ffc183","Type":"ContainerStarted","Data":"c61d2dfe0db31831349161d06880f7c33b9fb6a3e18d50020729bab442e3fbdf"} Jul 29 07:16:31.358254 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.358130 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:16:31.359688 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.359658 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d" exitCode=0 Jul 29 07:16:31.359811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.359741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d"} Jul 29 07:16:31.361482 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.361419 2565 generic.go:356] "Generic (PLEG): container finished" podID="7177cd99-fdf1-49b1-9ca2-70739ef9c4de" containerID="a4d4beda4f039e3eb6bf995d50f456df800aafda82813ce854c25ed4e048cf41" exitCode=0 Jul 29 07:16:31.361626 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.361551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5br72" event={"ID":"7177cd99-fdf1-49b1-9ca2-70739ef9c4de","Type":"ContainerDied","Data":"a4d4beda4f039e3eb6bf995d50f456df800aafda82813ce854c25ed4e048cf41"} Jul 29 07:16:31.363003 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.362979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" event={"ID":"b2453c3d-97fb-45f2-8062-cb27f05b8717","Type":"ContainerStarted","Data":"abc2147ac9e7611b31cd6e6565918dffde26467a3996ec7a1df7850e7ed6b13d"} Jul 29 07:16:31.364565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.364546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vhxbk" event={"ID":"f31a0c1a-5717-41ba-a99f-b218218bbbf6","Type":"ContainerStarted","Data":"b9b6f853ea61c21a8a7517a0ef81e49aac415c90b6f90d01e74aca11234dd46f"} Jul 29 07:16:31.376448 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.376406 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pj9lt" podStartSLOduration=66.544955877 podStartE2EDuration="1m12.376393291s" totalImagesPullingTime="5.831437414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:31.375471768 +0000 UTC m=+72.981649901" watchObservedRunningTime="2026-07-29 07:16:31.376393291 +0000 UTC m=+72.982571390" Jul 29 07:16:31.415640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.415600 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vhxbk" podStartSLOduration=66.464138768 podStartE2EDuration="1m12.415578945s" totalImagesPullingTime="5.951440177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:31.415032087 +0000 UTC m=+73.021210195" watchObservedRunningTime="2026-07-29 07:16:31.415578945 +0000 UTC m=+73.021757031" Jul 29 07:16:31.587177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.587146 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6566f4bc7b-jhq6f"] Jul 29 07:16:31.589317 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.589294 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.592693 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.592669 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 07:16:31.593009 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.592988 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 07:16:31.593124 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.592988 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 07:16:31.593298 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.593283 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-26zrp\"" Jul 29 07:16:31.593421 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.593401 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 07:16:31.594618 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.594598 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 07:16:31.594743 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.594638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 07:16:31.595315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.595296 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 07:16:31.601234 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.601218 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 07:16:31.609008 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.608911 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6566f4bc7b-jhq6f"] Jul 29 07:16:31.677561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-oauth-config\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.677722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5wp\" (UniqueName: \"kubernetes.io/projected/6257424d-f819-4507-9473-519831246a07-kube-api-access-qz5wp\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.677722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-trusted-ca-bundle\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.677722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-service-ca\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.677722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-serving-cert\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.677921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-console-config\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.677921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.677867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-oauth-serving-cert\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.778911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.778884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5wp\" (UniqueName: \"kubernetes.io/projected/6257424d-f819-4507-9473-519831246a07-kube-api-access-qz5wp\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.778919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-trusted-ca-bundle\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.778948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-service-ca\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.778995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-serving-cert\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.779029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-console-config\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.779115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-oauth-serving-cert\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.779149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-oauth-config\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.779927 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.779902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-console-config\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.780046 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.779926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-service-ca\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.780046 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.779945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-oauth-serving-cert\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.780125 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.780064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-trusted-ca-bundle\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.781848 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.781799 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-oauth-config\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.781993 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.781969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-serving-cert\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.788078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.788055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5wp\" (UniqueName: \"kubernetes.io/projected/6257424d-f819-4507-9473-519831246a07-kube-api-access-qz5wp\") pod \"console-6566f4bc7b-jhq6f\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:31.900249 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:31.900188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:32.041441 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:32.041408 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6566f4bc7b-jhq6f"] Jul 29 07:16:32.373141 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:32.372934 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5br72" event={"ID":"7177cd99-fdf1-49b1-9ca2-70739ef9c4de","Type":"ContainerStarted","Data":"d4d19cc08d8321beaec54aea6495f845b0a94db1e149ce689cca20fd8260a081"} Jul 29 07:16:32.373141 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:32.373032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5br72" event={"ID":"7177cd99-fdf1-49b1-9ca2-70739ef9c4de","Type":"ContainerStarted","Data":"6711eca4d0202f1586c26a9d0d3a007da805c57bcd0e7b2ad24be82b41532f52"} Jul 29 07:16:32.405027 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:32.404618 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5br72" podStartSLOduration=3.504382916 podStartE2EDuration="8.404596134s" totalImagesPullingTime="4.825472653s" totalInitContainerRuntime="74.740565ms" isStatefulPod=true podCreationTimestamp="2026-07-29 07:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:32.402568621 +0000 UTC m=+74.008746731" watchObservedRunningTime="2026-07-29 07:16:32.404596134 +0000 UTC m=+74.010774243" Jul 29 07:16:32.539976 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:32.539933 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6257424d_f819_4507_9473_519831246a07.slice/crio-581ce3f2fa42be3d385a3df3aca695a8f386463dfcf3d84d9ab08faa2fa66176 WatchSource:0}: Error finding container 581ce3f2fa42be3d385a3df3aca695a8f386463dfcf3d84d9ab08faa2fa66176: Status 404 returned error can't find the container with id 581ce3f2fa42be3d385a3df3aca695a8f386463dfcf3d84d9ab08faa2fa66176 Jul 29 07:16:33.377376 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:33.377350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6566f4bc7b-jhq6f" event={"ID":"6257424d-f819-4507-9473-519831246a07","Type":"ContainerStarted","Data":"581ce3f2fa42be3d385a3df3aca695a8f386463dfcf3d84d9ab08faa2fa66176"} Jul 29 07:16:34.385587 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.385557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6"} Jul 29 07:16:34.385928 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.385650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193"} Jul 29 07:16:34.385928 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.385688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f"} Jul 29 07:16:34.385928 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.385717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c"} Jul 29 07:16:34.385928 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.385749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1"} Jul 29 07:16:34.388975 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.388936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" event={"ID":"b2453c3d-97fb-45f2-8062-cb27f05b8717","Type":"ContainerStarted","Data":"0e1e4643982b122aabe2998df0975237fff65b01a6b10e7a62e578197bea0947"} Jul 29 07:16:34.389082 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.388998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" event={"ID":"b2453c3d-97fb-45f2-8062-cb27f05b8717","Type":"ContainerStarted","Data":"8fdbb48b6457b82c733678335cab6ca5518452789b59b91cb1f9ec7f8e435521"} Jul 29 07:16:34.389082 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.389024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" event={"ID":"b2453c3d-97fb-45f2-8062-cb27f05b8717","Type":"ContainerStarted","Data":"4cdc04cb872122bf1cdbdccfef80b7ef8affe998ffdaf2441f4c39e709da3daf"} Jul 29 07:16:34.391649 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.391624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"82e881f44714c73dd534bd263690e6d7597bb6120f8a502e0b0fcc2b4c311d4f"} Jul 29 07:16:34.391744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.391695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"5fe1db80d371bdc1eea4741f59b78ffa8d84e6e61c9b766d0d13a83fe05a69c1"} Jul 29 07:16:34.391744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.391731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"0b8116cff486d3b8fd54f60d1dcf1297fc527ef2caf9c858670a36b7353eeab4"} Jul 29 07:16:34.427181 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:34.427114 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56d4bf98f7-fw9z7" podStartSLOduration=3.145009979 podStartE2EDuration="5.427093047s" totalImagesPullingTime="2.282083068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:34.425025184 +0000 UTC m=+76.031203293" watchObservedRunningTime="2026-07-29 07:16:34.427093047 +0000 UTC m=+76.033271156" Jul 29 07:16:35.401557 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.401524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerStarted","Data":"2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a"} Jul 29 07:16:35.404862 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.404820 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"990ee17dc44a00ae89a8260709c8eb0e191671d11fbe6f7e5189bcea7caf0a21"} Jul 29 07:16:35.405012 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.404902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"613e60289282a41fae8a31cc847f3260b8917ae973cb79d7c45dce85862294d6"} Jul 29 07:16:35.405012 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.404946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" event={"ID":"19e3cb62-307a-40af-b497-3afb406dc032","Type":"ContainerStarted","Data":"8a7e0ee7725428aacbd4056ab3bb2468ab9a1919d2c094d728ab3db356a129ab"} Jul 29 07:16:35.453585 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.453515 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.005753917 podStartE2EDuration="10.453495841s" totalImagesPullingTime="6.428462348s" totalInitContainerRuntime="19.279576ms" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:35.45339063 +0000 UTC m=+77.059568739" watchObservedRunningTime="2026-07-29 07:16:35.453495841 +0000 UTC m=+77.059673951" Jul 29 07:16:35.480435 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.480407 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6566f4bc7b-jhq6f"] Jul 29 07:16:35.484525 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.484469 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" podStartSLOduration=2.910688445 podStartE2EDuration="8.484450358s" totalImagesPullingTime="5.573761913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:35.483062826 +0000 UTC m=+77.089240936" watchObservedRunningTime="2026-07-29 07:16:35.484450358 +0000 UTC m=+77.090628468" Jul 29 07:16:35.517985 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.517945 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d87747bf9-b6f59"] Jul 29 07:16:35.521385 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.521363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.533500 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.533466 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d87747bf9-b6f59"] Jul 29 07:16:35.615333 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-config\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.615518 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-service-ca\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.615518 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615502 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-oauth-config\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.615625 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-trusted-ca-bundle\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.615672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615627 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fg8\" (UniqueName: \"kubernetes.io/projected/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-kube-api-access-d6fg8\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.615672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-serving-cert\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.615759 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.615690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-oauth-serving-cert\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716182 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-service-ca\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716182 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-oauth-config\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-trusted-ca-bundle\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fg8\" (UniqueName: \"kubernetes.io/projected/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-kube-api-access-d6fg8\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-serving-cert\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-oauth-serving-cert\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-config\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.716941 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716924 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-config\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.717016 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-service-ca\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.717016 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.716975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-oauth-serving-cert\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.717117 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.717061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-trusted-ca-bundle\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.719477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.719451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-oauth-config\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.719606 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.719537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-serving-cert\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.724760 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.724742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fg8\" (UniqueName: \"kubernetes.io/projected/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-kube-api-access-d6fg8\") pod \"console-5d87747bf9-b6f59\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:35.832529 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:35.832495 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:36.065589 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.065563 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d87747bf9-b6f59"] Jul 29 07:16:36.067653 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:16:36.067625 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170eb8a9_3a9b_4c6b_8f75_3ff50e0e67ff.slice/crio-b6ae52fe83e589f0015dbd26b8e013d70e642062275700b7ea3abe3925ee4a00 WatchSource:0}: Error finding container b6ae52fe83e589f0015dbd26b8e013d70e642062275700b7ea3abe3925ee4a00: Status 404 returned error can't find the container with id b6ae52fe83e589f0015dbd26b8e013d70e642062275700b7ea3abe3925ee4a00 Jul 29 07:16:36.410217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.410183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6566f4bc7b-jhq6f" event={"ID":"6257424d-f819-4507-9473-519831246a07","Type":"ContainerStarted","Data":"074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7"} Jul 29 07:16:36.411901 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.411869 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d87747bf9-b6f59" event={"ID":"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff","Type":"ContainerStarted","Data":"0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3"} Jul 29 07:16:36.412092 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.411917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d87747bf9-b6f59" event={"ID":"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff","Type":"ContainerStarted","Data":"b6ae52fe83e589f0015dbd26b8e013d70e642062275700b7ea3abe3925ee4a00"} Jul 29 07:16:36.412156 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.412135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:36.436778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.436729 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6566f4bc7b-jhq6f" podStartSLOduration=1.997326081 podStartE2EDuration="5.436691114s" totalImagesPullingTime="3.439365033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:36.435683736 +0000 UTC m=+78.041861847" watchObservedRunningTime="2026-07-29 07:16:36.436691114 +0000 UTC m=+78.042869221" Jul 29 07:16:36.459114 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:36.458985 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d87747bf9-b6f59" podStartSLOduration=1.458976091 podStartE2EDuration="1.458976091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:16:36.457772067 +0000 UTC m=+78.063950180" watchObservedRunningTime="2026-07-29 07:16:36.458976091 +0000 UTC m=+78.065154198" Jul 29 07:16:41.900430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:41.900313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:16:42.420972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:42.420921 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6898c4698c-gl8g2" Jul 29 07:16:45.833339 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:45.833312 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:45.833846 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:45.833588 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:45.838851 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:45.838823 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:16:46.449995 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:16:46.449947 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:17:01.429011 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.428926 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6566f4bc7b-jhq6f" podUID="6257424d-f819-4507-9473-519831246a07" containerName="console" containerID="cri-o://074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7" gracePeriod=15 Jul 29 07:17:01.669896 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.669874 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:17:01.813674 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813649 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-oauth-serving-cert\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.813844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813709 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-service-ca\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.813844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-oauth-config\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.813844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813751 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-serving-cert\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.813844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813773 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz5wp\" (UniqueName: \"kubernetes.io/projected/6257424d-f819-4507-9473-519831246a07-kube-api-access-qz5wp\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.813844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813796 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-console-config\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.813844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.813825 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-trusted-ca-bundle\") pod \"6257424d-f819-4507-9473-519831246a07\" (UID: \"6257424d-f819-4507-9473-519831246a07\") " Jul 29 07:17:01.814243 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.814179 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:17:01.814316 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.814264 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-console-config" (OuterVolumeSpecName: "console-config") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:17:01.814316 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.814294 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-service-ca" (OuterVolumeSpecName: "service-ca") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:17:01.814391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.814336 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:17:01.815984 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.815943 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6257424d-f819-4507-9473-519831246a07-kube-api-access-qz5wp" (OuterVolumeSpecName: "kube-api-access-qz5wp") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "kube-api-access-qz5wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:17:01.816299 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.816279 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:01.816361 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.816314 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6257424d-f819-4507-9473-519831246a07" (UID: "6257424d-f819-4507-9473-519831246a07"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:01.914589 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914559 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-trusted-ca-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:01.914589 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914583 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-oauth-serving-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:01.914589 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914594 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:01.914793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914603 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-oauth-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:01.914793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914612 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6257424d-f819-4507-9473-519831246a07-console-serving-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:01.914793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914621 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz5wp\" (UniqueName: \"kubernetes.io/projected/6257424d-f819-4507-9473-519831246a07-kube-api-access-qz5wp\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:01.914793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:01.914630 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6257424d-f819-4507-9473-519831246a07-console-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:02.375626 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.375594 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pj9lt" Jul 29 07:17:02.509948 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.509916 2565 generic.go:356] "Generic (PLEG): container finished" podID="6257424d-f819-4507-9473-519831246a07" containerID="074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7" exitCode=0 Jul 29 07:17:02.510411 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.509976 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6566f4bc7b-jhq6f" Jul 29 07:17:02.510411 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.510004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6566f4bc7b-jhq6f" event={"ID":"6257424d-f819-4507-9473-519831246a07","Type":"ContainerDied","Data":"074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7"} Jul 29 07:17:02.510411 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.510068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6566f4bc7b-jhq6f" event={"ID":"6257424d-f819-4507-9473-519831246a07","Type":"ContainerDied","Data":"581ce3f2fa42be3d385a3df3aca695a8f386463dfcf3d84d9ab08faa2fa66176"} Jul 29 07:17:02.510411 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.510103 2565 scope.go:117] "RemoveContainer" containerID="074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7" Jul 29 07:17:02.518793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.518764 2565 scope.go:117] "RemoveContainer" containerID="074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7" Jul 29 07:17:02.519118 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:02.519091 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7\": container with ID starting with 074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7 not found: ID does not exist" containerID="074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7" Jul 29 07:17:02.519191 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.519126 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7"} err="failed to get container status \"074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7\": rpc error: code = NotFound desc = could not find container \"074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7\": container with ID starting with 074c0fd5b89aea98f1b2433ad7924031b81bc7787ace4623d8719944b0b704f7 not found: ID does not exist" Jul 29 07:17:02.531966 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.531927 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6566f4bc7b-jhq6f"] Jul 29 07:17:02.535034 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:02.535012 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6566f4bc7b-jhq6f"] Jul 29 07:17:03.018413 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:03.018381 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6257424d-f819-4507-9473-519831246a07" path="/var/lib/kubelet/pods/6257424d-f819-4507-9473-519831246a07/volumes" Jul 29 07:17:41.485557 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.485522 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c697c4754-4hsrk"] Jul 29 07:17:41.485945 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.485789 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6257424d-f819-4507-9473-519831246a07" containerName="console" Jul 29 07:17:41.485945 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.485802 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6257424d-f819-4507-9473-519831246a07" containerName="console" Jul 29 07:17:41.485945 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.485861 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6257424d-f819-4507-9473-519831246a07" containerName="console" Jul 29 07:17:41.487763 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.487744 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.503908 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.503886 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c697c4754-4hsrk"] Jul 29 07:17:41.629271 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-console-config\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.629271 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-oauth-config\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.629477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvd8r\" (UniqueName: \"kubernetes.io/projected/7f780598-0017-4302-a8a7-2d042f864fbd-kube-api-access-wvd8r\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.629477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-serving-cert\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.629477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-oauth-serving-cert\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.629571 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-trusted-ca-bundle\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.629571 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.629524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-service-ca\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.730707 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-service-ca\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.730707 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-console-config\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.730920 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-oauth-config\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.730920 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvd8r\" (UniqueName: \"kubernetes.io/projected/7f780598-0017-4302-a8a7-2d042f864fbd-kube-api-access-wvd8r\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.730920 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-serving-cert\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.731088 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-oauth-serving-cert\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.731088 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.730974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-trusted-ca-bundle\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.731518 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.731494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-service-ca\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.731622 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.731523 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-console-config\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.731779 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.731754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-oauth-serving-cert\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.731856 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.731796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-trusted-ca-bundle\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.733349 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.733328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-serving-cert\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.733425 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.733357 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-oauth-config\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.741401 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.741355 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvd8r\" (UniqueName: \"kubernetes.io/projected/7f780598-0017-4302-a8a7-2d042f864fbd-kube-api-access-wvd8r\") pod \"console-5c697c4754-4hsrk\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.796608 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.796581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:41.918547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:41.918524 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c697c4754-4hsrk"] Jul 29 07:17:41.920485 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:17:41.920461 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f780598_0017_4302_a8a7_2d042f864fbd.slice/crio-cf43a09ba89ebbc02301e4ee8b9a2ef447557d6a52776a4fc5127eb93f076ab7 WatchSource:0}: Error finding container cf43a09ba89ebbc02301e4ee8b9a2ef447557d6a52776a4fc5127eb93f076ab7: Status 404 returned error can't find the container with id cf43a09ba89ebbc02301e4ee8b9a2ef447557d6a52776a4fc5127eb93f076ab7 Jul 29 07:17:42.655813 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:42.655777 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c697c4754-4hsrk" event={"ID":"7f780598-0017-4302-a8a7-2d042f864fbd","Type":"ContainerStarted","Data":"f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf"} Jul 29 07:17:42.656309 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:42.655832 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c697c4754-4hsrk" event={"ID":"7f780598-0017-4302-a8a7-2d042f864fbd","Type":"ContainerStarted","Data":"cf43a09ba89ebbc02301e4ee8b9a2ef447557d6a52776a4fc5127eb93f076ab7"} Jul 29 07:17:42.678564 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:42.676559 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c697c4754-4hsrk" podStartSLOduration=1.676540519 podStartE2EDuration="1.676540519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:17:42.673935394 +0000 UTC m=+144.280113503" watchObservedRunningTime="2026-07-29 07:17:42.676540519 +0000 UTC m=+144.282718630" Jul 29 07:17:44.663247 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663214 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:17:44.663738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663489 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="alertmanager" containerID="cri-o://59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1" gracePeriod=120 Jul 29 07:17:44.663738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663555 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-metric" containerID="cri-o://d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6" gracePeriod=120 Jul 29 07:17:44.663738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663580 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="prom-label-proxy" containerID="cri-o://2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a" gracePeriod=120 Jul 29 07:17:44.663738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663618 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="config-reloader" containerID="cri-o://e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c" gracePeriod=120 Jul 29 07:17:44.663738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663625 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy" containerID="cri-o://91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193" gracePeriod=120 Jul 29 07:17:44.663738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:44.663619 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-web" containerID="cri-o://0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f" gracePeriod=120 Jul 29 07:17:45.671388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671357 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a" exitCode=0 Jul 29 07:17:45.671388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671382 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6" exitCode=0 Jul 29 07:17:45.671388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671389 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193" exitCode=0 Jul 29 07:17:45.671388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671396 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c" exitCode=0 Jul 29 07:17:45.671843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671402 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1" exitCode=0 Jul 29 07:17:45.671843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671432 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a"} Jul 29 07:17:45.671843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6"} Jul 29 07:17:45.671843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193"} Jul 29 07:17:45.671843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671550 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c"} Jul 29 07:17:45.671843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.671575 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1"} Jul 29 07:17:45.911139 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.911116 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:45.969849 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.969777 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-main-db\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.969849 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.969819 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-web\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970047 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.969856 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-main-tls\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970047 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.969892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970047 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.969924 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q75s8\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-kube-api-access-q75s8\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970255 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970223 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:17:45.970391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-metrics-client-ca\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970391 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970382 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-out\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970343 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:17:45.970547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970489 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:17:45.970689 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970416 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-tls-assets\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970749 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970710 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970749 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970745 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-web-config\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970770 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-volume\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-cluster-tls-config\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.970854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.970847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy\") pod \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\" (UID: \"d7c6a7fe-29c4-4891-8fd2-11f11d767b69\") " Jul 29 07:17:45.971179 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.971161 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-main-db\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:45.971346 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.971333 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:45.971442 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.971429 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-metrics-client-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:45.973725 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.973651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:45.973973 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.973834 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:17:45.974388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.974362 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:45.974627 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.974598 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-out" (OuterVolumeSpecName: "config-out") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:17:45.974627 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.974613 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-kube-api-access-q75s8" (OuterVolumeSpecName: "kube-api-access-q75s8") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "kube-api-access-q75s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:17:45.974770 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.974632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:45.974770 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.974702 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:45.974876 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.974818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-volume" (OuterVolumeSpecName: "config-volume") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:45.978534 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.978505 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:45.985851 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:45.985827 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-web-config" (OuterVolumeSpecName: "web-config") pod "d7c6a7fe-29c4-4891-8fd2-11f11d767b69" (UID: "d7c6a7fe-29c4-4891-8fd2-11f11d767b69"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:17:46.072970 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.072928 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-out\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.072970 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.072977 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-tls-assets\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073199 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.072991 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073199 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073004 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-web-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073687 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073642 2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073687 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073682 2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-cluster-tls-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073706 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073729 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073759 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-secret-alertmanager-main-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.073844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.073779 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75s8\" (UniqueName: \"kubernetes.io/projected/d7c6a7fe-29c4-4891-8fd2-11f11d767b69-kube-api-access-q75s8\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:17:46.677997 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.677934 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerID="0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f" exitCode=0 Jul 29 07:17:46.678477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.677991 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f"} Jul 29 07:17:46.678477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.678057 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.678477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.678117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7c6a7fe-29c4-4891-8fd2-11f11d767b69","Type":"ContainerDied","Data":"ce74c56762ad55a3684f402e5e4786d277955bcfb21a0e29040c95fb4d8d4d28"} Jul 29 07:17:46.678477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.678174 2565 scope.go:117] "RemoveContainer" containerID="2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a" Jul 29 07:17:46.686011 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.685993 2565 scope.go:117] "RemoveContainer" containerID="d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6" Jul 29 07:17:46.692765 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.692748 2565 scope.go:117] "RemoveContainer" containerID="91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193" Jul 29 07:17:46.699065 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.699049 2565 scope.go:117] "RemoveContainer" containerID="0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f" Jul 29 07:17:46.702294 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.702271 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:17:46.706190 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.706173 2565 scope.go:117] "RemoveContainer" containerID="e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c" Jul 29 07:17:46.707913 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.707893 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:17:46.712827 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.712812 2565 scope.go:117] "RemoveContainer" containerID="59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1" Jul 29 07:17:46.719471 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.719443 2565 scope.go:117] "RemoveContainer" containerID="9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d" Jul 29 07:17:46.725521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.725506 2565 scope.go:117] "RemoveContainer" containerID="2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a" Jul 29 07:17:46.725761 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.725742 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a\": container with ID starting with 2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a not found: ID does not exist" containerID="2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a" Jul 29 07:17:46.725810 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.725772 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a"} err="failed to get container status \"2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a\": rpc error: code = NotFound desc = could not find container \"2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a\": container with ID starting with 2dcc97de57aeaa9ab0fd3da568e85e96d36e462398356e1c7119a5aaee7b5b9a not found: ID does not exist" Jul 29 07:17:46.725810 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.725794 2565 scope.go:117] "RemoveContainer" containerID="d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6" Jul 29 07:17:46.726029 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.726001 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6\": container with ID starting with d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6 not found: ID does not exist" containerID="d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6" Jul 29 07:17:46.726157 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726037 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6"} err="failed to get container status \"d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6\": rpc error: code = NotFound desc = could not find container \"d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6\": container with ID starting with d768cc30c0a6624231cd2ee26d18e15f8d38555960ee77fe3ae5e2ec170d15b6 not found: ID does not exist" Jul 29 07:17:46.726157 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726060 2565 scope.go:117] "RemoveContainer" containerID="91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193" Jul 29 07:17:46.726342 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.726322 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193\": container with ID starting with 91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193 not found: ID does not exist" containerID="91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193" Jul 29 07:17:46.726410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726346 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193"} err="failed to get container status \"91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193\": rpc error: code = NotFound desc = could not find container \"91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193\": container with ID starting with 91442e3e719c1ce99bcfa964e2ae186dcb83e97508af0bebd6bba198e1bae193 not found: ID does not exist" Jul 29 07:17:46.726410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726361 2565 scope.go:117] "RemoveContainer" containerID="0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f" Jul 29 07:17:46.726599 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.726580 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f\": container with ID starting with 0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f not found: ID does not exist" containerID="0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f" Jul 29 07:17:46.726649 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726604 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f"} err="failed to get container status \"0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f\": rpc error: code = NotFound desc = could not find container \"0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f\": container with ID starting with 0af12a0374bfbbd50572a9df8a62521f688b39ed4c8cac8368c7644e178ae08f not found: ID does not exist" Jul 29 07:17:46.726649 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726619 2565 scope.go:117] "RemoveContainer" containerID="e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c" Jul 29 07:17:46.726852 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.726832 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c\": container with ID starting with e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c not found: ID does not exist" containerID="e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c" Jul 29 07:17:46.726897 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726858 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c"} err="failed to get container status \"e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c\": rpc error: code = NotFound desc = could not find container \"e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c\": container with ID starting with e7ea6a685755ea3b2643beb6889b71c67aa056a1e1ff98b32a870b65ec057e9c not found: ID does not exist" Jul 29 07:17:46.726897 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.726876 2565 scope.go:117] "RemoveContainer" containerID="59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1" Jul 29 07:17:46.727156 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.727138 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1\": container with ID starting with 59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1 not found: ID does not exist" containerID="59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1" Jul 29 07:17:46.727212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.727158 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1"} err="failed to get container status \"59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1\": rpc error: code = NotFound desc = could not find container \"59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1\": container with ID starting with 59dd8b768b321b4115c878d7e40a8103ffc1854cb7942013c76db7c166ff9ca1 not found: ID does not exist" Jul 29 07:17:46.727212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.727173 2565 scope.go:117] "RemoveContainer" containerID="9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d" Jul 29 07:17:46.727456 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:17:46.727439 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d\": container with ID starting with 9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d not found: ID does not exist" containerID="9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d" Jul 29 07:17:46.727509 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.727460 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d"} err="failed to get container status \"9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d\": rpc error: code = NotFound desc = could not find container \"9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d\": container with ID starting with 9d15dc4261c694ada34a96430675f5d2336addc7f2d37cf7ce408b865b90f25d not found: ID does not exist" Jul 29 07:17:46.735168 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735146 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:17:46.735414 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735400 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="alertmanager" Jul 29 07:17:46.735451 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735416 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="alertmanager" Jul 29 07:17:46.735451 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735432 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-web" Jul 29 07:17:46.735451 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735437 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-web" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735453 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="prom-label-proxy" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735460 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="prom-label-proxy" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735466 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="init-config-reloader" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735473 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="init-config-reloader" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735486 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="config-reloader" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735494 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="config-reloader" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735503 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735511 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735517 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-metric" Jul 29 07:17:46.735538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735522 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-metric" Jul 29 07:17:46.735809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735561 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="alertmanager" Jul 29 07:17:46.735809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735569 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-web" Jul 29 07:17:46.735809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735577 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy" Jul 29 07:17:46.735809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735587 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="prom-label-proxy" Jul 29 07:17:46.735809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735595 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="kube-rbac-proxy-metric" Jul 29 07:17:46.735809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.735601 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" containerName="config-reloader" Jul 29 07:17:46.739213 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.739197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.743774 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.743760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ddxxn\"" Jul 29 07:17:46.744122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744102 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 07:17:46.744222 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744166 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 07:17:46.744278 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744224 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 07:17:46.744465 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 07:17:46.744601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 07:17:46.744601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744555 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 07:17:46.744744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744613 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 07:17:46.744786 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.744764 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 07:17:46.749439 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.749422 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 07:17:46.754088 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.754070 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:17:46.881180 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-web-config\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-config-out\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrht\" (UniqueName: \"kubernetes.io/projected/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-kube-api-access-hvrht\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.881861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.881588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982372 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982372 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982372 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-web-config\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-config-out\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrht\" (UniqueName: \"kubernetes.io/projected/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-kube-api-access-hvrht\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.982636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.983157 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.982975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.983845 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.983506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.983845 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.983761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.985497 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.985467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.985598 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.985578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.985650 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.985578 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-web-config\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.985650 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.985579 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.985763 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.985737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.986096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.986075 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-config-out\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.986626 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.986605 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.986747 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.986724 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.987557 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.987535 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:46.991792 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:46.991769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrht\" (UniqueName: \"kubernetes.io/projected/cbfab147-2e2f-42fb-a0e9-2d684911c1b4-kube-api-access-hvrht\") pod \"alertmanager-main-0\" (UID: \"cbfab147-2e2f-42fb-a0e9-2d684911c1b4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:47.018126 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:47.018103 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c6a7fe-29c4-4891-8fd2-11f11d767b69" path="/var/lib/kubelet/pods/d7c6a7fe-29c4-4891-8fd2-11f11d767b69/volumes" Jul 29 07:17:47.048550 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:47.048532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 07:17:47.180022 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:47.179995 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 07:17:47.181711 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:17:47.181686 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfab147_2e2f_42fb_a0e9_2d684911c1b4.slice/crio-adc72d9bc3bd969bf60cdd15ffb5101b04334fd23b07a4a0a873a142427b98cd WatchSource:0}: Error finding container adc72d9bc3bd969bf60cdd15ffb5101b04334fd23b07a4a0a873a142427b98cd: Status 404 returned error can't find the container with id adc72d9bc3bd969bf60cdd15ffb5101b04334fd23b07a4a0a873a142427b98cd Jul 29 07:17:47.686251 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:47.686217 2565 generic.go:356] "Generic (PLEG): container finished" podID="cbfab147-2e2f-42fb-a0e9-2d684911c1b4" containerID="f856fea0360051f7fa71a8dcedf6b77e7719c48800a5779b809117fdac0b1e4b" exitCode=0 Jul 29 07:17:47.686623 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:47.686302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerDied","Data":"f856fea0360051f7fa71a8dcedf6b77e7719c48800a5779b809117fdac0b1e4b"} Jul 29 07:17:47.686623 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:47.686393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"adc72d9bc3bd969bf60cdd15ffb5101b04334fd23b07a4a0a873a142427b98cd"} Jul 29 07:17:48.693685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.693650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"e3beac7e45293d56413b151d728706b6443acfbb9f091e4612a83f10d06a0a37"} Jul 29 07:17:48.694177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.693725 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"7e3105bbb3b494793ec4e2b0e8682e950604b96dd89428cb8a55afd59fd3f97e"} Jul 29 07:17:48.694177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.693766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"2cdf6e760da22805112bdfc26ba5cf431c8359b8473a46caed8577570819ce76"} Jul 29 07:17:48.694177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.693813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"c4a0db3d87580df9e4f28b0a800a09a4fe679e00a79a74e4a8860e85ac1ac000"} Jul 29 07:17:48.694177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.693854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"b5f22e737a84c188d3c9fa70e9c9ed3ef02e9b04eb54288734f67dd5237f2739"} Jul 29 07:17:48.694177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.693890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbfab147-2e2f-42fb-a0e9-2d684911c1b4","Type":"ContainerStarted","Data":"6a1d2e06e47a3c706aad70d5f9131e7bdf13941e6edadaaaf8b457ff49656902"} Jul 29 07:17:48.724178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:48.724130 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.708538402 podStartE2EDuration="2.724114134s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.575732ms" isStatefulPod=false podCreationTimestamp="2026-07-29 07:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:17:48.721973614 +0000 UTC m=+150.328151713" watchObservedRunningTime="2026-07-29 07:17:48.724114134 +0000 UTC m=+150.330292273" Jul 29 07:17:51.796802 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:51.796753 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:51.797247 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:51.796824 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:51.801551 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:51.801530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:52.714304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:52.714278 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:17:52.758232 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:17:52.758200 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d87747bf9-b6f59"] Jul 29 07:18:17.780702 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.780646 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d87747bf9-b6f59" podUID="170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" containerName="console" containerID="cri-o://0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3" gracePeriod=15 Jul 29 07:18:17.867291 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.867263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c"] Jul 29 07:18:17.869567 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.869551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:17.872470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.872447 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 07:18:17.872470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.872460 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 07:18:17.873773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.873760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pdplf\"" Jul 29 07:18:17.879653 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.879634 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c"] Jul 29 07:18:17.932547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.932510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:17.932674 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.932566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:17.932715 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:17.932671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxql\" (UniqueName: \"kubernetes.io/projected/c031b9de-0fb1-45cc-a891-a376ac1c2a72-kube-api-access-qvxql\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.008156 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.008136 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:18:18.033555 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033494 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6fg8\" (UniqueName: \"kubernetes.io/projected/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-kube-api-access-d6fg8\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033542 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-oauth-config\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-serving-cert\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033597 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-trusted-ca-bundle\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033630 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-service-ca\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033863 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033673 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-oauth-serving-cert\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033863 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033740 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-config\") pod \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\" (UID: \"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff\") " Jul 29 07:18:18.033989 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.033989 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.033937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.034122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.034028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxql\" (UniqueName: \"kubernetes.io/projected/c031b9de-0fb1-45cc-a891-a376ac1c2a72-kube-api-access-qvxql\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.034122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.034073 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:18:18.034122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.034098 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-service-ca" (OuterVolumeSpecName: "service-ca") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:18:18.034832 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.034784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.034943 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.034830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-config" (OuterVolumeSpecName: "console-config") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:18:18.034943 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.034857 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.035162 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.035141 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:18:18.036020 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.035994 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:18:18.036185 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.036162 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:18:18.036185 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.036169 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-kube-api-access-d6fg8" (OuterVolumeSpecName: "kube-api-access-d6fg8") pod "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" (UID: "170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff"). InnerVolumeSpecName "kube-api-access-d6fg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:18:18.042722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.042696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxql\" (UniqueName: \"kubernetes.io/projected/c031b9de-0fb1-45cc-a891-a376ac1c2a72-kube-api-access-qvxql\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.134737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134713 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fg8\" (UniqueName: \"kubernetes.io/projected/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-kube-api-access-d6fg8\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.134737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134734 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-oauth-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.134865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134742 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-serving-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.134865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134751 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-trusted-ca-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.134865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134760 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.134865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134768 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-oauth-serving-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.134865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.134776 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff-console-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:18.177629 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.177607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:18.296790 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.296766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c"] Jul 29 07:18:18.299078 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:18:18.299058 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc031b9de_0fb1_45cc_a891_a376ac1c2a72.slice/crio-ff0da206520e93f8346287ac3dc6b5293697894a1f837f1c797970f1376d1374 WatchSource:0}: Error finding container ff0da206520e93f8346287ac3dc6b5293697894a1f837f1c797970f1376d1374: Status 404 returned error can't find the container with id ff0da206520e93f8346287ac3dc6b5293697894a1f837f1c797970f1376d1374 Jul 29 07:18:18.806716 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.806688 2565 generic.go:356] "Generic (PLEG): container finished" podID="170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" containerID="0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3" exitCode=0 Jul 29 07:18:18.807169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.806750 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d87747bf9-b6f59" Jul 29 07:18:18.807169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.806777 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d87747bf9-b6f59" event={"ID":"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff","Type":"ContainerDied","Data":"0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3"} Jul 29 07:18:18.807169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.806842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d87747bf9-b6f59" event={"ID":"170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff","Type":"ContainerDied","Data":"b6ae52fe83e589f0015dbd26b8e013d70e642062275700b7ea3abe3925ee4a00"} Jul 29 07:18:18.807169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.806871 2565 scope.go:117] "RemoveContainer" containerID="0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3" Jul 29 07:18:18.808228 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.807902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" event={"ID":"c031b9de-0fb1-45cc-a891-a376ac1c2a72","Type":"ContainerStarted","Data":"ff0da206520e93f8346287ac3dc6b5293697894a1f837f1c797970f1376d1374"} Jul 29 07:18:18.815066 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.815050 2565 scope.go:117] "RemoveContainer" containerID="0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3" Jul 29 07:18:18.815298 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:18:18.815279 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3\": container with ID starting with 0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3 not found: ID does not exist" containerID="0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3" Jul 29 07:18:18.815358 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.815305 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3"} err="failed to get container status \"0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3\": rpc error: code = NotFound desc = could not find container \"0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3\": container with ID starting with 0af6319cd1c3a96e24ff84e8e7569f92e16f0db863d92b985955ab26391909e3 not found: ID does not exist" Jul 29 07:18:18.827784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.827763 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d87747bf9-b6f59"] Jul 29 07:18:18.831822 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:18.831801 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d87747bf9-b6f59"] Jul 29 07:18:19.019316 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:19.019288 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" path="/var/lib/kubelet/pods/170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff/volumes" Jul 29 07:18:24.834558 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:24.834489 2565 generic.go:356] "Generic (PLEG): container finished" podID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerID="3a0e10ea32459a6eb10e6fc24718cd6c4b404e35fe59f97d042680b46cf42a89" exitCode=0 Jul 29 07:18:24.834558 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:24.834528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" event={"ID":"c031b9de-0fb1-45cc-a891-a376ac1c2a72","Type":"ContainerDied","Data":"3a0e10ea32459a6eb10e6fc24718cd6c4b404e35fe59f97d042680b46cf42a89"} Jul 29 07:18:26.849986 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:26.849904 2565 generic.go:356] "Generic (PLEG): container finished" podID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerID="a47abdf38359cbf3627e185629467a3c0d71efd9538aedbdcaed1558343c5709" exitCode=0 Jul 29 07:18:26.850277 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:26.849997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" event={"ID":"c031b9de-0fb1-45cc-a891-a376ac1c2a72","Type":"ContainerDied","Data":"a47abdf38359cbf3627e185629467a3c0d71efd9538aedbdcaed1558343c5709"} Jul 29 07:18:33.878834 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:33.878800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" event={"ID":"c031b9de-0fb1-45cc-a891-a376ac1c2a72","Type":"ContainerStarted","Data":"845d5341a5e4de41c48b52b4a397dfee305703e5f9a84c03ae213b375baab0b6"} Jul 29 07:18:33.898840 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:33.898796 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" podStartSLOduration=1.957330927 podStartE2EDuration="16.898780905s" totalImagesPullingTime="14.941449978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:33.896454371 +0000 UTC m=+195.502632479" watchObservedRunningTime="2026-07-29 07:18:33.898780905 +0000 UTC m=+195.504959075" Jul 29 07:18:34.884970 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:34.884926 2565 generic.go:356] "Generic (PLEG): container finished" podID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerID="845d5341a5e4de41c48b52b4a397dfee305703e5f9a84c03ae213b375baab0b6" exitCode=0 Jul 29 07:18:34.885346 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:34.885000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" event={"ID":"c031b9de-0fb1-45cc-a891-a376ac1c2a72","Type":"ContainerDied","Data":"845d5341a5e4de41c48b52b4a397dfee305703e5f9a84c03ae213b375baab0b6"} Jul 29 07:18:36.009125 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.009095 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:36.083796 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.083765 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-util\") pod \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " Jul 29 07:18:36.083971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.083828 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-bundle\") pod \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " Jul 29 07:18:36.083971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.083886 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvxql\" (UniqueName: \"kubernetes.io/projected/c031b9de-0fb1-45cc-a891-a376ac1c2a72-kube-api-access-qvxql\") pod \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\" (UID: \"c031b9de-0fb1-45cc-a891-a376ac1c2a72\") " Jul 29 07:18:36.084508 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.084476 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-bundle" (OuterVolumeSpecName: "bundle") pod "c031b9de-0fb1-45cc-a891-a376ac1c2a72" (UID: "c031b9de-0fb1-45cc-a891-a376ac1c2a72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:36.086083 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.086061 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c031b9de-0fb1-45cc-a891-a376ac1c2a72-kube-api-access-qvxql" (OuterVolumeSpecName: "kube-api-access-qvxql") pod "c031b9de-0fb1-45cc-a891-a376ac1c2a72" (UID: "c031b9de-0fb1-45cc-a891-a376ac1c2a72"). InnerVolumeSpecName "kube-api-access-qvxql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:18:36.088644 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.088620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-util" (OuterVolumeSpecName: "util") pod "c031b9de-0fb1-45cc-a891-a376ac1c2a72" (UID: "c031b9de-0fb1-45cc-a891-a376ac1c2a72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:36.184965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.184898 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvxql\" (UniqueName: \"kubernetes.io/projected/c031b9de-0fb1-45cc-a891-a376ac1c2a72-kube-api-access-qvxql\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:36.184965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.184920 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:36.184965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.184929 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c031b9de-0fb1-45cc-a891-a376ac1c2a72-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:36.893765 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.893732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" event={"ID":"c031b9de-0fb1-45cc-a891-a376ac1c2a72","Type":"ContainerDied","Data":"ff0da206520e93f8346287ac3dc6b5293697894a1f837f1c797970f1376d1374"} Jul 29 07:18:36.893915 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.893777 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0da206520e93f8346287ac3dc6b5293697894a1f837f1c797970f1376d1374" Jul 29 07:18:36.893915 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:36.893778 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkx5c" Jul 29 07:18:41.010008 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.009975 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x"] Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010261 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="pull" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010272 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="pull" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010290 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="extract" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010295 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="extract" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010303 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" containerName="console" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010308 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" containerName="console" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010314 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="util" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010319 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="util" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010374 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="170eb8a9-3a9b-4c6b-8f75-3ff50e0e67ff" containerName="console" Jul 29 07:18:41.010428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.010381 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c031b9de-0fb1-45cc-a891-a376ac1c2a72" containerName="extract" Jul 29 07:18:41.041217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.041195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x"] Jul 29 07:18:41.041335 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.041264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.044352 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.044245 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:18:41.044352 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.044323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p5g79\"" Jul 29 07:18:41.044352 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.044351 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 29 07:18:41.121586 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.121561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpl6\" (UniqueName: \"kubernetes.io/projected/b0154a90-f3bf-444f-8d60-a40782c9893c-kube-api-access-mlpl6\") pod \"cert-manager-operator-controller-manager-55c444bf86-dqz4x\" (UID: \"b0154a90-f3bf-444f-8d60-a40782c9893c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.121703 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.121634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0154a90-f3bf-444f-8d60-a40782c9893c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dqz4x\" (UID: \"b0154a90-f3bf-444f-8d60-a40782c9893c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.222738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.222714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0154a90-f3bf-444f-8d60-a40782c9893c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dqz4x\" (UID: \"b0154a90-f3bf-444f-8d60-a40782c9893c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.222848 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.222762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpl6\" (UniqueName: \"kubernetes.io/projected/b0154a90-f3bf-444f-8d60-a40782c9893c-kube-api-access-mlpl6\") pod \"cert-manager-operator-controller-manager-55c444bf86-dqz4x\" (UID: \"b0154a90-f3bf-444f-8d60-a40782c9893c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.223096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.223080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0154a90-f3bf-444f-8d60-a40782c9893c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dqz4x\" (UID: \"b0154a90-f3bf-444f-8d60-a40782c9893c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.231788 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.231771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpl6\" (UniqueName: \"kubernetes.io/projected/b0154a90-f3bf-444f-8d60-a40782c9893c-kube-api-access-mlpl6\") pod \"cert-manager-operator-controller-manager-55c444bf86-dqz4x\" (UID: \"b0154a90-f3bf-444f-8d60-a40782c9893c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.351057 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.351032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" Jul 29 07:18:41.505380 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.505354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x"] Jul 29 07:18:41.507864 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:18:41.507830 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0154a90_f3bf_444f_8d60_a40782c9893c.slice/crio-14123e74cb5a5c37b7f35241aaefbe18e99723553e9623acfee17e21cf1309cb WatchSource:0}: Error finding container 14123e74cb5a5c37b7f35241aaefbe18e99723553e9623acfee17e21cf1309cb: Status 404 returned error can't find the container with id 14123e74cb5a5c37b7f35241aaefbe18e99723553e9623acfee17e21cf1309cb Jul 29 07:18:41.913727 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:41.913690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" event={"ID":"b0154a90-f3bf-444f-8d60-a40782c9893c","Type":"ContainerStarted","Data":"14123e74cb5a5c37b7f35241aaefbe18e99723553e9623acfee17e21cf1309cb"} Jul 29 07:18:43.923591 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:43.923515 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" event={"ID":"b0154a90-f3bf-444f-8d60-a40782c9893c","Type":"ContainerStarted","Data":"b9074eb070b5ba5cd6180dd514f7e01235f3404d8d4cd6d0e27cc26db67692c9"} Jul 29 07:18:43.945928 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:43.945878 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dqz4x" podStartSLOduration=1.8918710330000001 podStartE2EDuration="3.945863201s" totalImagesPullingTime="2.053992168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:43.945228773 +0000 UTC m=+205.551406884" watchObservedRunningTime="2026-07-29 07:18:43.945863201 +0000 UTC m=+205.552041308" Jul 29 07:18:45.408047 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.408013 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h9hhl"] Jul 29 07:18:45.426044 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.426020 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h9hhl"] Jul 29 07:18:45.426186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.426113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.428969 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.428929 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 07:18:45.430164 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.430145 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 07:18:45.430299 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.430163 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4cbr6\"" Jul 29 07:18:45.555774 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.555748 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpcpf\" (UniqueName: \"kubernetes.io/projected/0b85b79a-bdea-4267-af21-b3d3a7d7a9ba-kube-api-access-jpcpf\") pod \"cert-manager-webhook-6486ffd997-h9hhl\" (UID: \"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.555866 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.555807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b85b79a-bdea-4267-af21-b3d3a7d7a9ba-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h9hhl\" (UID: \"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.656726 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.656704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b85b79a-bdea-4267-af21-b3d3a7d7a9ba-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h9hhl\" (UID: \"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.656825 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.656752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpcpf\" (UniqueName: \"kubernetes.io/projected/0b85b79a-bdea-4267-af21-b3d3a7d7a9ba-kube-api-access-jpcpf\") pod \"cert-manager-webhook-6486ffd997-h9hhl\" (UID: \"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.667162 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.667110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b85b79a-bdea-4267-af21-b3d3a7d7a9ba-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h9hhl\" (UID: \"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.667289 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.667272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpcpf\" (UniqueName: \"kubernetes.io/projected/0b85b79a-bdea-4267-af21-b3d3a7d7a9ba-kube-api-access-jpcpf\") pod \"cert-manager-webhook-6486ffd997-h9hhl\" (UID: \"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.690007 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.689986 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c"] Jul 29 07:18:45.710828 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.710806 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c"] Jul 29 07:18:45.710907 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.710892 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.713636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.713607 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 07:18:45.713777 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.713753 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 07:18:45.713878 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.713827 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pdplf\"" Jul 29 07:18:45.747344 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.747323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:45.858524 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.858490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.858663 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.858537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.858663 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.858570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxxk\" (UniqueName: \"kubernetes.io/projected/b57481d3-4e73-4d32-89a0-f4960e212e07-kube-api-access-6hxxk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.887237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.887211 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h9hhl"] Jul 29 07:18:45.889709 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:18:45.889685 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b85b79a_bdea_4267_af21_b3d3a7d7a9ba.slice/crio-4a4534e292cec871ea85cd15a2eb6f75747baca4db54bd7caae2c07d9551f71b WatchSource:0}: Error finding container 4a4534e292cec871ea85cd15a2eb6f75747baca4db54bd7caae2c07d9551f71b: Status 404 returned error can't find the container with id 4a4534e292cec871ea85cd15a2eb6f75747baca4db54bd7caae2c07d9551f71b Jul 29 07:18:45.931548 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.931490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" event={"ID":"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba","Type":"ContainerStarted","Data":"4a4534e292cec871ea85cd15a2eb6f75747baca4db54bd7caae2c07d9551f71b"} Jul 29 07:18:45.959320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.959300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.959408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.959346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.959408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.959374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxxk\" (UniqueName: \"kubernetes.io/projected/b57481d3-4e73-4d32-89a0-f4960e212e07-kube-api-access-6hxxk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.959720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.959698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.959759 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.959706 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:45.967325 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:45.967303 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxxk\" (UniqueName: \"kubernetes.io/projected/b57481d3-4e73-4d32-89a0-f4960e212e07-kube-api-access-6hxxk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:46.020512 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:46.020490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:46.170846 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:46.168881 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c"] Jul 29 07:18:46.937865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:46.937827 2565 generic.go:356] "Generic (PLEG): container finished" podID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerID="b11babfba5a636e6e246f09d18778a07e99b77d491bb30fa3e235db63dea30cd" exitCode=0 Jul 29 07:18:46.938297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:46.937910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" event={"ID":"b57481d3-4e73-4d32-89a0-f4960e212e07","Type":"ContainerDied","Data":"b11babfba5a636e6e246f09d18778a07e99b77d491bb30fa3e235db63dea30cd"} Jul 29 07:18:46.938297 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:46.937981 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" event={"ID":"b57481d3-4e73-4d32-89a0-f4960e212e07","Type":"ContainerStarted","Data":"0ca079aea66f3c030fe89c97609757090ee65ecf465f328e6b1c9435079826c1"} Jul 29 07:18:49.954987 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:49.954935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" event={"ID":"0b85b79a-bdea-4267-af21-b3d3a7d7a9ba","Type":"ContainerStarted","Data":"d1a340e232fdaa08ea13b096f48bbfeb3c3709f4422c1a8639242c834d7915bd"} Jul 29 07:18:49.955524 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:49.955011 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:49.956778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:49.956754 2565 generic.go:356] "Generic (PLEG): container finished" podID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerID="7a7d611270143618c0f7b622c84edcce6d42c85331b3ccb2e5753525440bbc80" exitCode=0 Jul 29 07:18:49.956891 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:49.956790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" event={"ID":"b57481d3-4e73-4d32-89a0-f4960e212e07","Type":"ContainerDied","Data":"7a7d611270143618c0f7b622c84edcce6d42c85331b3ccb2e5753525440bbc80"} Jul 29 07:18:49.971791 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:49.971748 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" podStartSLOduration=1.512606906 podStartE2EDuration="4.9717344s" totalImagesPullingTime="3.459127494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:49.971162448 +0000 UTC m=+211.577340548" watchObservedRunningTime="2026-07-29 07:18:49.9717344 +0000 UTC m=+211.577912508" Jul 29 07:18:50.962784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:50.962748 2565 generic.go:356] "Generic (PLEG): container finished" podID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerID="7424c2ad34810a0d21337cbb2df0b48444a5b45ee91864eda81a3d14836c5f31" exitCode=0 Jul 29 07:18:50.963179 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:50.962837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" event={"ID":"b57481d3-4e73-4d32-89a0-f4960e212e07","Type":"ContainerDied","Data":"7424c2ad34810a0d21337cbb2df0b48444a5b45ee91864eda81a3d14836c5f31"} Jul 29 07:18:52.082746 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.082720 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:52.213839 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.213811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-bundle\") pod \"b57481d3-4e73-4d32-89a0-f4960e212e07\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " Jul 29 07:18:52.213984 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.213850 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxxk\" (UniqueName: \"kubernetes.io/projected/b57481d3-4e73-4d32-89a0-f4960e212e07-kube-api-access-6hxxk\") pod \"b57481d3-4e73-4d32-89a0-f4960e212e07\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " Jul 29 07:18:52.213984 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.213878 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-util\") pod \"b57481d3-4e73-4d32-89a0-f4960e212e07\" (UID: \"b57481d3-4e73-4d32-89a0-f4960e212e07\") " Jul 29 07:18:52.214267 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.214232 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-bundle" (OuterVolumeSpecName: "bundle") pod "b57481d3-4e73-4d32-89a0-f4960e212e07" (UID: "b57481d3-4e73-4d32-89a0-f4960e212e07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:52.215883 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.215862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57481d3-4e73-4d32-89a0-f4960e212e07-kube-api-access-6hxxk" (OuterVolumeSpecName: "kube-api-access-6hxxk") pod "b57481d3-4e73-4d32-89a0-f4960e212e07" (UID: "b57481d3-4e73-4d32-89a0-f4960e212e07"). InnerVolumeSpecName "kube-api-access-6hxxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:18:52.219458 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.219410 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-util" (OuterVolumeSpecName: "util") pod "b57481d3-4e73-4d32-89a0-f4960e212e07" (UID: "b57481d3-4e73-4d32-89a0-f4960e212e07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:18:52.314569 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.314542 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:52.314569 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.314567 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxxk\" (UniqueName: \"kubernetes.io/projected/b57481d3-4e73-4d32-89a0-f4960e212e07-kube-api-access-6hxxk\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:52.314718 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.314578 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b57481d3-4e73-4d32-89a0-f4960e212e07-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:18:52.972661 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.972623 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" event={"ID":"b57481d3-4e73-4d32-89a0-f4960e212e07","Type":"ContainerDied","Data":"0ca079aea66f3c030fe89c97609757090ee65ecf465f328e6b1c9435079826c1"} Jul 29 07:18:52.972661 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.972661 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftpz9c" Jul 29 07:18:52.972848 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:52.972674 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca079aea66f3c030fe89c97609757090ee65ecf465f328e6b1c9435079826c1" Jul 29 07:18:55.965112 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:55.965084 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h9hhl" Jul 29 07:18:57.064043 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064014 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lfp4d"] Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064287 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="pull" Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064297 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="pull" Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064307 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="util" Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064313 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="util" Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064330 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="extract" Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064335 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="extract" Jul 29 07:18:57.064396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.064378 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57481d3-4e73-4d32-89a0-f4960e212e07" containerName="extract" Jul 29 07:18:57.066284 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.066267 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.069046 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.069023 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jtbgj\"" Jul 29 07:18:57.074926 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.074897 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lfp4d"] Jul 29 07:18:57.148610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.148556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c755f54-97b0-47bd-8ea3-134ccfabf321-bound-sa-token\") pod \"cert-manager-55b4d954d5-lfp4d\" (UID: \"8c755f54-97b0-47bd-8ea3-134ccfabf321\") " pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.148610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.148607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7ls\" (UniqueName: \"kubernetes.io/projected/8c755f54-97b0-47bd-8ea3-134ccfabf321-kube-api-access-6m7ls\") pod \"cert-manager-55b4d954d5-lfp4d\" (UID: \"8c755f54-97b0-47bd-8ea3-134ccfabf321\") " pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.249840 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.249809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c755f54-97b0-47bd-8ea3-134ccfabf321-bound-sa-token\") pod \"cert-manager-55b4d954d5-lfp4d\" (UID: \"8c755f54-97b0-47bd-8ea3-134ccfabf321\") " pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.249840 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.249839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7ls\" (UniqueName: \"kubernetes.io/projected/8c755f54-97b0-47bd-8ea3-134ccfabf321-kube-api-access-6m7ls\") pod \"cert-manager-55b4d954d5-lfp4d\" (UID: \"8c755f54-97b0-47bd-8ea3-134ccfabf321\") " pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.258638 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.258615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c755f54-97b0-47bd-8ea3-134ccfabf321-bound-sa-token\") pod \"cert-manager-55b4d954d5-lfp4d\" (UID: \"8c755f54-97b0-47bd-8ea3-134ccfabf321\") " pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.258851 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.258830 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7ls\" (UniqueName: \"kubernetes.io/projected/8c755f54-97b0-47bd-8ea3-134ccfabf321-kube-api-access-6m7ls\") pod \"cert-manager-55b4d954d5-lfp4d\" (UID: \"8c755f54-97b0-47bd-8ea3-134ccfabf321\") " pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.375752 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.375680 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lfp4d" Jul 29 07:18:57.498429 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.498358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lfp4d"] Jul 29 07:18:57.500370 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:18:57.500343 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c755f54_97b0_47bd_8ea3_134ccfabf321.slice/crio-8e73fe90d7ead052776607e782ad92111382573123e028c0c476ef8f2095b438 WatchSource:0}: Error finding container 8e73fe90d7ead052776607e782ad92111382573123e028c0c476ef8f2095b438: Status 404 returned error can't find the container with id 8e73fe90d7ead052776607e782ad92111382573123e028c0c476ef8f2095b438 Jul 29 07:18:57.993823 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.993783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lfp4d" event={"ID":"8c755f54-97b0-47bd-8ea3-134ccfabf321","Type":"ContainerStarted","Data":"657e18a0c2b8bc86377a2b043d2cc1a3ddc95c11926c8eecb8f81f4af5d1282c"} Jul 29 07:18:57.994000 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:57.993832 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lfp4d" event={"ID":"8c755f54-97b0-47bd-8ea3-134ccfabf321","Type":"ContainerStarted","Data":"8e73fe90d7ead052776607e782ad92111382573123e028c0c476ef8f2095b438"} Jul 29 07:18:58.010777 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:18:58.010734 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lfp4d" podStartSLOduration=1.010719428 podStartE2EDuration="1.010719428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:58.009969677 +0000 UTC m=+219.616147777" watchObservedRunningTime="2026-07-29 07:18:58.010719428 +0000 UTC m=+219.616897536" Jul 29 07:19:04.363438 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.363402 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2"] Jul 29 07:19:04.366139 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.366116 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.369056 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.369025 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pdplf\"" Jul 29 07:19:04.369197 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.369089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 07:19:04.370267 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.370250 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 07:19:04.375277 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.375251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2"] Jul 29 07:19:04.508042 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.508021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.508164 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.508068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznwv\" (UniqueName: \"kubernetes.io/projected/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-kube-api-access-vznwv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.508164 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.508113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.609314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.609289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.609440 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.609379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.609489 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.609458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznwv\" (UniqueName: \"kubernetes.io/projected/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-kube-api-access-vznwv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.609622 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.609607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.609663 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.609638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.617401 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.617342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznwv\" (UniqueName: \"kubernetes.io/projected/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-kube-api-access-vznwv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.676585 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.676558 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:04.799210 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:04.799184 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2"] Jul 29 07:19:04.800937 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:19:04.800908 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d5f587_b10d_441a_8f84_1bbdc25af2fc.slice/crio-6174980590aac4795546a631b167c69d1084b4dccf507f8b2b4a5b48de4a74f0 WatchSource:0}: Error finding container 6174980590aac4795546a631b167c69d1084b4dccf507f8b2b4a5b48de4a74f0: Status 404 returned error can't find the container with id 6174980590aac4795546a631b167c69d1084b4dccf507f8b2b4a5b48de4a74f0 Jul 29 07:19:05.029171 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:05.029130 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerID="fc6b4798e0636a26228cdbc4c3f1c5ba34f94e0800816ee1506dac71c3341f47" exitCode=0 Jul 29 07:19:05.029334 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:05.029193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" event={"ID":"a3d5f587-b10d-441a-8f84-1bbdc25af2fc","Type":"ContainerDied","Data":"fc6b4798e0636a26228cdbc4c3f1c5ba34f94e0800816ee1506dac71c3341f47"} Jul 29 07:19:05.029334 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:05.029269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" event={"ID":"a3d5f587-b10d-441a-8f84-1bbdc25af2fc","Type":"ContainerStarted","Data":"6174980590aac4795546a631b167c69d1084b4dccf507f8b2b4a5b48de4a74f0"} Jul 29 07:19:06.035235 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:06.035151 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerID="03c1dc87e1e12a2995c1a67fb99d213c55a37e1f601db6b68683c22bfd40a40d" exitCode=0 Jul 29 07:19:06.035235 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:06.035204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" event={"ID":"a3d5f587-b10d-441a-8f84-1bbdc25af2fc","Type":"ContainerDied","Data":"03c1dc87e1e12a2995c1a67fb99d213c55a37e1f601db6b68683c22bfd40a40d"} Jul 29 07:19:07.041094 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:07.041063 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerID="80855467a4cb260815c3392b19ae7c07eec391af4ef056600257b288a65f7d95" exitCode=0 Jul 29 07:19:07.041452 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:07.041152 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" event={"ID":"a3d5f587-b10d-441a-8f84-1bbdc25af2fc","Type":"ContainerDied","Data":"80855467a4cb260815c3392b19ae7c07eec391af4ef056600257b288a65f7d95"} Jul 29 07:19:08.169290 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.169266 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:08.236614 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.236592 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vznwv\" (UniqueName: \"kubernetes.io/projected/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-kube-api-access-vznwv\") pod \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " Jul 29 07:19:08.236743 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.236620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-bundle\") pod \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " Jul 29 07:19:08.236743 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.236653 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-util\") pod \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\" (UID: \"a3d5f587-b10d-441a-8f84-1bbdc25af2fc\") " Jul 29 07:19:08.237280 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.237260 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-bundle" (OuterVolumeSpecName: "bundle") pod "a3d5f587-b10d-441a-8f84-1bbdc25af2fc" (UID: "a3d5f587-b10d-441a-8f84-1bbdc25af2fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:19:08.238620 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.238594 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-kube-api-access-vznwv" (OuterVolumeSpecName: "kube-api-access-vznwv") pod "a3d5f587-b10d-441a-8f84-1bbdc25af2fc" (UID: "a3d5f587-b10d-441a-8f84-1bbdc25af2fc"). InnerVolumeSpecName "kube-api-access-vznwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:19:08.242424 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.242327 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-util" (OuterVolumeSpecName: "util") pod "a3d5f587-b10d-441a-8f84-1bbdc25af2fc" (UID: "a3d5f587-b10d-441a-8f84-1bbdc25af2fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:19:08.337258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.337227 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznwv\" (UniqueName: \"kubernetes.io/projected/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-kube-api-access-vznwv\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:19:08.337258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.337258 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:19:08.337258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:08.337267 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3d5f587-b10d-441a-8f84-1bbdc25af2fc-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:19:09.050212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:09.050183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" Jul 29 07:19:09.050347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:09.050208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af66wv2" event={"ID":"a3d5f587-b10d-441a-8f84-1bbdc25af2fc","Type":"ContainerDied","Data":"6174980590aac4795546a631b167c69d1084b4dccf507f8b2b4a5b48de4a74f0"} Jul 29 07:19:09.050347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:09.050252 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6174980590aac4795546a631b167c69d1084b4dccf507f8b2b4a5b48de4a74f0" Jul 29 07:19:19.066055 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066022 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d85576cc8-79swl"] Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066300 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="util" Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066311 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="util" Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066330 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="pull" Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066335 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="pull" Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066344 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="extract" Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066349 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="extract" Jul 29 07:19:19.066408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.066395 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d5f587-b10d-441a-8f84-1bbdc25af2fc" containerName="extract" Jul 29 07:19:19.086927 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.086896 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d85576cc8-79swl"] Jul 29 07:19:19.087089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.087021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.091561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.091508 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 07:19:19.091561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.091539 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 07:19:19.091784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.091593 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 07:19:19.091784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.091510 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 07:19:19.091784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.091676 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g4gdz\"" Jul 29 07:19:19.091784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.091754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 07:19:19.114177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.114153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc077e91-8114-48e1-906c-ee9473f47196-cert\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.114341 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.114200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc077e91-8114-48e1-906c-ee9473f47196-metrics-cert\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.114341 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.114256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc077e91-8114-48e1-906c-ee9473f47196-manager-config\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.114341 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.114315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtlwk\" (UniqueName: \"kubernetes.io/projected/dc077e91-8114-48e1-906c-ee9473f47196-kube-api-access-jtlwk\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.215036 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.214999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc077e91-8114-48e1-906c-ee9473f47196-cert\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.215220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.215048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc077e91-8114-48e1-906c-ee9473f47196-metrics-cert\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.215220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.215088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc077e91-8114-48e1-906c-ee9473f47196-manager-config\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.215220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.215131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtlwk\" (UniqueName: \"kubernetes.io/projected/dc077e91-8114-48e1-906c-ee9473f47196-kube-api-access-jtlwk\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.215915 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.215892 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc077e91-8114-48e1-906c-ee9473f47196-manager-config\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.217529 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.217508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc077e91-8114-48e1-906c-ee9473f47196-cert\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.217609 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.217563 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc077e91-8114-48e1-906c-ee9473f47196-metrics-cert\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.230642 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.230619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtlwk\" (UniqueName: \"kubernetes.io/projected/dc077e91-8114-48e1-906c-ee9473f47196-kube-api-access-jtlwk\") pod \"lws-controller-manager-d85576cc8-79swl\" (UID: \"dc077e91-8114-48e1-906c-ee9473f47196\") " pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.239093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.239070 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps"] Jul 29 07:19:19.248605 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.248587 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.252102 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.252068 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 07:19:19.252282 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.252260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 07:19:19.252382 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.252362 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sjl6m\"" Jul 29 07:19:19.252382 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.252374 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 07:19:19.252734 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.252713 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 07:19:19.265009 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.264985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps"] Jul 29 07:19:19.316534 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.316449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qthj\" (UniqueName: \"kubernetes.io/projected/a8e2f522-b621-4c5f-b340-8d81eea2dc37-kube-api-access-6qthj\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.316534 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.316493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8e2f522-b621-4c5f-b340-8d81eea2dc37-manifests\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.316534 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.316519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e2f522-b621-4c5f-b340-8d81eea2dc37-tmp\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.316731 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.316573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8e2f522-b621-4c5f-b340-8d81eea2dc37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.316731 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.316599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8e2f522-b621-4c5f-b340-8d81eea2dc37-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.397408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.397375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:19.419225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.417853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qthj\" (UniqueName: \"kubernetes.io/projected/a8e2f522-b621-4c5f-b340-8d81eea2dc37-kube-api-access-6qthj\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.419225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.417908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8e2f522-b621-4c5f-b340-8d81eea2dc37-manifests\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.419225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.417943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e2f522-b621-4c5f-b340-8d81eea2dc37-tmp\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.419225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.418016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8e2f522-b621-4c5f-b340-8d81eea2dc37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.419225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.418043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8e2f522-b621-4c5f-b340-8d81eea2dc37-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.419877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.419851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a8e2f522-b621-4c5f-b340-8d81eea2dc37-manifests\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.421270 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.421243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8e2f522-b621-4c5f-b340-8d81eea2dc37-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.421651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.421627 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8e2f522-b621-4c5f-b340-8d81eea2dc37-tmp\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.422060 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.422038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a8e2f522-b621-4c5f-b340-8d81eea2dc37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.428048 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.428020 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qthj\" (UniqueName: \"kubernetes.io/projected/a8e2f522-b621-4c5f-b340-8d81eea2dc37-kube-api-access-6qthj\") pod \"opendatahub-operator-controller-manager-6d6c966467-9c4ps\" (UID: \"a8e2f522-b621-4c5f-b340-8d81eea2dc37\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.559384 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.559350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:19.566992 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.566915 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d85576cc8-79swl"] Jul 29 07:19:19.570215 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:19:19.570177 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc077e91_8114_48e1_906c_ee9473f47196.slice/crio-fc87c99fda5a713ac3772dd1ab649fd97a3fa8c00e41e76c187dee6bb8a10ae5 WatchSource:0}: Error finding container fc87c99fda5a713ac3772dd1ab649fd97a3fa8c00e41e76c187dee6bb8a10ae5: Status 404 returned error can't find the container with id fc87c99fda5a713ac3772dd1ab649fd97a3fa8c00e41e76c187dee6bb8a10ae5 Jul 29 07:19:19.720691 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:19.720663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps"] Jul 29 07:19:19.723276 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:19:19.723245 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e2f522_b621_4c5f_b340_8d81eea2dc37.slice/crio-7882c9d8d72a0bc1c7dc64b19646edaefd80981cf923ec7c61d3a7d869833967 WatchSource:0}: Error finding container 7882c9d8d72a0bc1c7dc64b19646edaefd80981cf923ec7c61d3a7d869833967: Status 404 returned error can't find the container with id 7882c9d8d72a0bc1c7dc64b19646edaefd80981cf923ec7c61d3a7d869833967 Jul 29 07:19:20.098018 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:20.097981 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" event={"ID":"dc077e91-8114-48e1-906c-ee9473f47196","Type":"ContainerStarted","Data":"fc87c99fda5a713ac3772dd1ab649fd97a3fa8c00e41e76c187dee6bb8a10ae5"} Jul 29 07:19:20.099258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:20.099234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" event={"ID":"a8e2f522-b621-4c5f-b340-8d81eea2dc37","Type":"ContainerStarted","Data":"7882c9d8d72a0bc1c7dc64b19646edaefd80981cf923ec7c61d3a7d869833967"} Jul 29 07:19:23.116544 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:23.116491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" event={"ID":"dc077e91-8114-48e1-906c-ee9473f47196","Type":"ContainerStarted","Data":"40ce1ce57d1ca6ed9afc5a7fd606f590cd37b00c5308188cf1f43f53732235d5"} Jul 29 07:19:23.116999 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:23.116567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:23.118107 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:23.118078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" event={"ID":"a8e2f522-b621-4c5f-b340-8d81eea2dc37","Type":"ContainerStarted","Data":"374ab43bdb46603633970ad72b2c4ab702b1411a6ab7bb70abe336d4eee42268"} Jul 29 07:19:23.138026 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:23.137923 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" podStartSLOduration=0.83051062 podStartE2EDuration="4.13790413s" totalImagesPullingTime="3.30739351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:23.136285341 +0000 UTC m=+244.742463449" watchObservedRunningTime="2026-07-29 07:19:23.13790413 +0000 UTC m=+244.744082240" Jul 29 07:19:24.124002 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:24.123971 2565 generic.go:356] "Generic (PLEG): container finished" podID="a8e2f522-b621-4c5f-b340-8d81eea2dc37" containerID="374ab43bdb46603633970ad72b2c4ab702b1411a6ab7bb70abe336d4eee42268" exitCode=0 Jul 29 07:19:24.124307 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:24.124048 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" event={"ID":"a8e2f522-b621-4c5f-b340-8d81eea2dc37","Type":"ContainerDied","Data":"374ab43bdb46603633970ad72b2c4ab702b1411a6ab7bb70abe336d4eee42268"} Jul 29 07:19:25.130291 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:25.130259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" event={"ID":"a8e2f522-b621-4c5f-b340-8d81eea2dc37","Type":"ContainerStarted","Data":"c344244b31bd09204fbd3a811142721c193bd73d6c65927d1d40659f292b8fa0"} Jul 29 07:19:25.130701 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:25.130356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:25.153143 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:25.153090 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" podStartSLOduration=1.5765471770000001 podStartE2EDuration="6.153071277s" totalImagesPullingTime="3.498558186s" totalInitContainerRuntime="1.077965914s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:25.150500611 +0000 UTC m=+246.756678733" watchObservedRunningTime="2026-07-29 07:19:25.153071277 +0000 UTC m=+246.759249386" Jul 29 07:19:34.126131 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:34.126100 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d85576cc8-79swl" Jul 29 07:19:36.137378 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:36.137346 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6c966467-9c4ps" Jul 29 07:19:46.326809 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.326721 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl"] Jul 29 07:19:46.330037 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.330015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.332922 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.332900 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 07:19:46.333108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.332917 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 07:19:46.333108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.332909 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 07:19:46.333108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.332900 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 07:19:46.333275 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.333186 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 07:19:46.333455 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.333432 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 07:19:46.333569 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.333487 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4dmts\"" Jul 29 07:19:46.342692 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.342669 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl"] Jul 29 07:19:46.452986 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.452938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97n4\" (UniqueName: \"kubernetes.io/projected/142741f2-587b-4bd3-8dd5-a514b4ea172b-kube-api-access-x97n4\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.452986 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.452987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.453178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.453114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.453178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.453144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.453178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.453163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.453277 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.453239 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/142741f2-587b-4bd3-8dd5-a514b4ea172b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.453311 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.453271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.554620 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.554580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.554620 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.554617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.554844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.554633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.555124 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.555099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/142741f2-587b-4bd3-8dd5-a514b4ea172b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.555236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.555151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.555236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.555201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97n4\" (UniqueName: \"kubernetes.io/projected/142741f2-587b-4bd3-8dd5-a514b4ea172b-kube-api-access-x97n4\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.555336 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.555236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.556824 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.556791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.557968 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.557926 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.564531 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.564507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.564638 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.564595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.564720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.564703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142741f2-587b-4bd3-8dd5-a514b4ea172b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.565561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.565545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/142741f2-587b-4bd3-8dd5-a514b4ea172b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.571886 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.571865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97n4\" (UniqueName: \"kubernetes.io/projected/142741f2-587b-4bd3-8dd5-a514b4ea172b-kube-api-access-x97n4\") pod \"istiod-openshift-gateway-5677ccb49b-jm2kl\" (UID: \"142741f2-587b-4bd3-8dd5-a514b4ea172b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.638943 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.638864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:46.774217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:46.774190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl"] Jul 29 07:19:46.776061 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:19:46.776035 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142741f2_587b_4bd3_8dd5_a514b4ea172b.slice/crio-f03793e8c467b0baa47fd7b1433e2949cd9bcbcba47795e36b8d4de145d39b07 WatchSource:0}: Error finding container f03793e8c467b0baa47fd7b1433e2949cd9bcbcba47795e36b8d4de145d39b07: Status 404 returned error can't find the container with id f03793e8c467b0baa47fd7b1433e2949cd9bcbcba47795e36b8d4de145d39b07 Jul 29 07:19:47.234531 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:47.234495 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" event={"ID":"142741f2-587b-4bd3-8dd5-a514b4ea172b","Type":"ContainerStarted","Data":"f03793e8c467b0baa47fd7b1433e2949cd9bcbcba47795e36b8d4de145d39b07"} Jul 29 07:19:48.084302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.084271 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps"] Jul 29 07:19:48.087034 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.087005 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.091832 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.091784 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qxbwd\"" Jul 29 07:19:48.106882 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.106858 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps"] Jul 29 07:19:48.168014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.167969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f32fe0-45b7-4605-bf8b-43370da88b1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168236 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwf8\" (UniqueName: \"kubernetes.io/projected/65f32fe0-45b7-4605-bf8b-43370da88b1c-kube-api-access-phwf8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168575 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168575 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.168575 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.168349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269629 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f32fe0-45b7-4605-bf8b-43370da88b1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.269877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwf8\" (UniqueName: \"kubernetes.io/projected/65f32fe0-45b7-4605-bf8b-43370da88b1c-kube-api-access-phwf8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.270285 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.269884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.270285 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.270209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.270285 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.270267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.270700 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.270656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f32fe0-45b7-4605-bf8b-43370da88b1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.270821 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.270722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.270821 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.270729 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.273190 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.273165 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.273486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.273464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.289102 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.289075 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwf8\" (UniqueName: \"kubernetes.io/projected/65f32fe0-45b7-4605-bf8b-43370da88b1c-kube-api-access-phwf8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.295089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.294993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f32fe0-45b7-4605-bf8b-43370da88b1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps\" (UID: \"65f32fe0-45b7-4605-bf8b-43370da88b1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.400048 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.399963 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:48.575363 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:48.575339 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps"] Jul 29 07:19:49.453270 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:49.453233 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:19:49.453569 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:49.453296 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:19:50.260207 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:50.260165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" event={"ID":"142741f2-587b-4bd3-8dd5-a514b4ea172b","Type":"ContainerStarted","Data":"8f903f4c988c83434adf4e46d745ccd951929af74ddddec84ee9b79dd8835cfe"} Jul 29 07:19:50.260394 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:50.260268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:50.261731 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:50.261674 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" event={"ID":"65f32fe0-45b7-4605-bf8b-43370da88b1c","Type":"ContainerStarted","Data":"c882cf26d5f28d5efedc263e490b446955888e8f757a7a4daaa317bbfc5eb404"} Jul 29 07:19:50.262030 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:50.262007 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-jm2kl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 07:19:50.262101 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:50.262052 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" podUID="142741f2-587b-4bd3-8dd5-a514b4ea172b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 07:19:50.316608 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:50.316554 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" podStartSLOduration=1.641517127 podStartE2EDuration="4.316538894s" totalImagesPullingTime="2.675021767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:50.313914928 +0000 UTC m=+271.920093071" watchObservedRunningTime="2026-07-29 07:19:50.316538894 +0000 UTC m=+271.922717004" Jul 29 07:19:51.267548 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:51.267522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jm2kl" Jul 29 07:19:52.146114 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:52.146079 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:19:52.146242 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:52.146159 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:19:52.146242 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:52.146205 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:19:52.274108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:52.274071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" event={"ID":"65f32fe0-45b7-4605-bf8b-43370da88b1c","Type":"ContainerStarted","Data":"ea9a6cf4e3cd994de7c8aa0e200afe4a1d21934b72d88f30542832bd7384ea1e"} Jul 29 07:19:52.296698 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:52.296646 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" podStartSLOduration=1.5668934719999998 podStartE2EDuration="4.296625598s" totalImagesPullingTime="2.729732126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:19:52.295325127 +0000 UTC m=+273.901503238" watchObservedRunningTime="2026-07-29 07:19:52.296625598 +0000 UTC m=+273.902803707" Jul 29 07:19:52.400579 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:52.400489 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:53.404869 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:53.404839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:54.282477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:54.282446 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:19:54.283374 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:19:54.283356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps" Jul 29 07:20:18.829701 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:18.829674 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 07:20:55.761437 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.761401 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2"] Jul 29 07:20:55.764032 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.764015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:55.767037 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.767009 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 07:20:55.767615 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.767602 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 07:20:55.768501 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.768483 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2l2dr\"" Jul 29 07:20:55.775330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.775308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2"] Jul 29 07:20:55.914741 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.914710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:55.914870 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.914750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:55.914870 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.914776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxdd\" (UniqueName: \"kubernetes.io/projected/aa36ba9a-0a57-4151-9e05-891e03cd76eb-kube-api-access-mgxdd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:55.961320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.961292 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r"] Jul 29 07:20:55.963827 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.963808 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:55.975794 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:55.975775 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r"] Jul 29 07:20:56.015353 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.015298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.015353 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.015330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.015517 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.015355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxdd\" (UniqueName: \"kubernetes.io/projected/aa36ba9a-0a57-4151-9e05-891e03cd76eb-kube-api-access-mgxdd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.015639 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.015619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.015708 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.015686 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.028282 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.028259 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxdd\" (UniqueName: \"kubernetes.io/projected/aa36ba9a-0a57-4151-9e05-891e03cd76eb-kube-api-access-mgxdd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.073338 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.073318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:20:56.116527 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.116500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.116669 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.116540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dld6w\" (UniqueName: \"kubernetes.io/projected/35ece30b-f78d-4175-83df-e665307b551c-kube-api-access-dld6w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.116734 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.116678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.204363 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.204339 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2"] Jul 29 07:20:56.206478 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:20:56.206447 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa36ba9a_0a57_4151_9e05_891e03cd76eb.slice/crio-1d29f56e7120dcfd15d86f867aafec4b52193eaea0a9931cbe7a66c32ac8692e WatchSource:0}: Error finding container 1d29f56e7120dcfd15d86f867aafec4b52193eaea0a9931cbe7a66c32ac8692e: Status 404 returned error can't find the container with id 1d29f56e7120dcfd15d86f867aafec4b52193eaea0a9931cbe7a66c32ac8692e Jul 29 07:20:56.217768 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.217747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dld6w\" (UniqueName: \"kubernetes.io/projected/35ece30b-f78d-4175-83df-e665307b551c-kube-api-access-dld6w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.217837 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.217801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.217875 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.217847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.218169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.218154 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.218212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.218192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.227900 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.227879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dld6w\" (UniqueName: \"kubernetes.io/projected/35ece30b-f78d-4175-83df-e665307b551c-kube-api-access-dld6w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.272658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.272613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:20:56.372580 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.372553 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb"] Jul 29 07:20:56.380503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.380477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.394066 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.394040 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb"] Jul 29 07:20:56.402431 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.402275 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r"] Jul 29 07:20:56.404308 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:20:56.404281 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ece30b_f78d_4175_83df_e665307b551c.slice/crio-ec76a506c0ee4e54db44b19fff91d5c724b47839999a55ad84570b9b0edf4c0c WatchSource:0}: Error finding container ec76a506c0ee4e54db44b19fff91d5c724b47839999a55ad84570b9b0edf4c0c: Status 404 returned error can't find the container with id ec76a506c0ee4e54db44b19fff91d5c724b47839999a55ad84570b9b0edf4c0c Jul 29 07:20:56.520364 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.520338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.520454 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.520386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.520496 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.520476 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqn8r\" (UniqueName: \"kubernetes.io/projected/c496b17e-3fe8-4094-916f-4ddfb88812f9-kube-api-access-rqn8r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.556744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.556712 2565 generic.go:356] "Generic (PLEG): container finished" podID="35ece30b-f78d-4175-83df-e665307b551c" containerID="88e2fe42167e801d0b6732367d0042b2f0003bd21caebbea0080b7e7004c029b" exitCode=0 Jul 29 07:20:56.556833 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.556796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" event={"ID":"35ece30b-f78d-4175-83df-e665307b551c","Type":"ContainerDied","Data":"88e2fe42167e801d0b6732367d0042b2f0003bd21caebbea0080b7e7004c029b"} Jul 29 07:20:56.556874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.556847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" event={"ID":"35ece30b-f78d-4175-83df-e665307b551c","Type":"ContainerStarted","Data":"ec76a506c0ee4e54db44b19fff91d5c724b47839999a55ad84570b9b0edf4c0c"} Jul 29 07:20:56.557329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.557314 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:20:56.558301 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.558283 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerID="8b1ac25ed7db6a9e2ba2b7db27f0f40264f10f68f580110c29fde02a0a1b3782" exitCode=0 Jul 29 07:20:56.558389 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.558325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" event={"ID":"aa36ba9a-0a57-4151-9e05-891e03cd76eb","Type":"ContainerDied","Data":"8b1ac25ed7db6a9e2ba2b7db27f0f40264f10f68f580110c29fde02a0a1b3782"} Jul 29 07:20:56.558389 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.558354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" event={"ID":"aa36ba9a-0a57-4151-9e05-891e03cd76eb","Type":"ContainerStarted","Data":"1d29f56e7120dcfd15d86f867aafec4b52193eaea0a9931cbe7a66c32ac8692e"} Jul 29 07:20:56.621903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.621877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.622007 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.621923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.622007 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.621969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqn8r\" (UniqueName: \"kubernetes.io/projected/c496b17e-3fe8-4094-916f-4ddfb88812f9-kube-api-access-rqn8r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.622321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.622298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.622362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.622315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.631888 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.631870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqn8r\" (UniqueName: \"kubernetes.io/projected/c496b17e-3fe8-4094-916f-4ddfb88812f9-kube-api-access-rqn8r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.690219 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.690198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:20:56.815310 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.815290 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb"] Jul 29 07:20:56.816126 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:20:56.816103 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc496b17e_3fe8_4094_916f_4ddfb88812f9.slice/crio-aad3b46bf05e1fbfb6996ea48f0b4d9382f8c70bf31982581f40dcd4ef6124f9 WatchSource:0}: Error finding container aad3b46bf05e1fbfb6996ea48f0b4d9382f8c70bf31982581f40dcd4ef6124f9: Status 404 returned error can't find the container with id aad3b46bf05e1fbfb6996ea48f0b4d9382f8c70bf31982581f40dcd4ef6124f9 Jul 29 07:20:56.961938 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.961907 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb"] Jul 29 07:20:56.964543 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.964529 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:56.981680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:56.981659 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb"] Jul 29 07:20:57.127173 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.127115 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.127286 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.127263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.127421 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.127403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmjx\" (UniqueName: \"kubernetes.io/projected/9d2d8b8b-30db-472a-9d44-2b9b54d44320-kube-api-access-jgmjx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.228061 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.228030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.228185 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.228092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmjx\" (UniqueName: \"kubernetes.io/projected/9d2d8b8b-30db-472a-9d44-2b9b54d44320-kube-api-access-jgmjx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.228185 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.228123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.228393 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.228376 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.228433 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.228410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.239633 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.239613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmjx\" (UniqueName: \"kubernetes.io/projected/9d2d8b8b-30db-472a-9d44-2b9b54d44320-kube-api-access-jgmjx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.274361 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.274340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:20:57.404812 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.404667 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb"] Jul 29 07:20:57.454297 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:20:57.454263 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2d8b8b_30db_472a_9d44_2b9b54d44320.slice/crio-764ac8197268149af226f9a88075b91f067a939ef6d308b017dccd92e698f764 WatchSource:0}: Error finding container 764ac8197268149af226f9a88075b91f067a939ef6d308b017dccd92e698f764: Status 404 returned error can't find the container with id 764ac8197268149af226f9a88075b91f067a939ef6d308b017dccd92e698f764 Jul 29 07:20:57.565773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.565729 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" event={"ID":"9d2d8b8b-30db-472a-9d44-2b9b54d44320","Type":"ContainerStarted","Data":"764ac8197268149af226f9a88075b91f067a939ef6d308b017dccd92e698f764"} Jul 29 07:20:57.567467 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.567442 2565 generic.go:356] "Generic (PLEG): container finished" podID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerID="bb229099d28dbe9b62b4405cc46010502f6c8082cbb0d0986638c8fc46e8951b" exitCode=0 Jul 29 07:20:57.567594 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.567521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" event={"ID":"c496b17e-3fe8-4094-916f-4ddfb88812f9","Type":"ContainerDied","Data":"bb229099d28dbe9b62b4405cc46010502f6c8082cbb0d0986638c8fc46e8951b"} Jul 29 07:20:57.567594 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:57.567570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" event={"ID":"c496b17e-3fe8-4094-916f-4ddfb88812f9","Type":"ContainerStarted","Data":"aad3b46bf05e1fbfb6996ea48f0b4d9382f8c70bf31982581f40dcd4ef6124f9"} Jul 29 07:20:58.574491 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.574416 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerID="15e5f5c64a98b030f9f6d7a172aec8a595e4b8671bc7990fc06565a62b6c01c2" exitCode=0 Jul 29 07:20:58.574877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.574488 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" event={"ID":"aa36ba9a-0a57-4151-9e05-891e03cd76eb","Type":"ContainerDied","Data":"15e5f5c64a98b030f9f6d7a172aec8a595e4b8671bc7990fc06565a62b6c01c2"} Jul 29 07:20:58.576108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.576083 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerID="5a969f17903034c94e4338ccce51a955cdd2b8fd8d700817101122baf704d06e" exitCode=0 Jul 29 07:20:58.576202 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.576171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" event={"ID":"9d2d8b8b-30db-472a-9d44-2b9b54d44320","Type":"ContainerDied","Data":"5a969f17903034c94e4338ccce51a955cdd2b8fd8d700817101122baf704d06e"} Jul 29 07:20:58.577913 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.577892 2565 generic.go:356] "Generic (PLEG): container finished" podID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerID="3897c69fdc40f35e05a341d70bd6b857c1601c6753001d5de64d7d924befe1a6" exitCode=0 Jul 29 07:20:58.578033 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.577966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" event={"ID":"c496b17e-3fe8-4094-916f-4ddfb88812f9","Type":"ContainerDied","Data":"3897c69fdc40f35e05a341d70bd6b857c1601c6753001d5de64d7d924befe1a6"} Jul 29 07:20:58.579976 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.579942 2565 generic.go:356] "Generic (PLEG): container finished" podID="35ece30b-f78d-4175-83df-e665307b551c" containerID="af0cc9a6e0777882b4838bcab5bb3a4122a4856f1d4f720e6bf02f1203c63bb6" exitCode=0 Jul 29 07:20:58.580083 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:58.579982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" event={"ID":"35ece30b-f78d-4175-83df-e665307b551c","Type":"ContainerDied","Data":"af0cc9a6e0777882b4838bcab5bb3a4122a4856f1d4f720e6bf02f1203c63bb6"} Jul 29 07:20:59.587191 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.587158 2565 generic.go:356] "Generic (PLEG): container finished" podID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerID="a0dcacf0b0cca85d894f688907c025b3e23828729244bb265df9cbe4a46ac6d6" exitCode=0 Jul 29 07:20:59.587544 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.587233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" event={"ID":"aa36ba9a-0a57-4151-9e05-891e03cd76eb","Type":"ContainerDied","Data":"a0dcacf0b0cca85d894f688907c025b3e23828729244bb265df9cbe4a46ac6d6"} Jul 29 07:20:59.588919 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.588900 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerID="89be277fe52c79f3e3c6f4c2ade6ad1d8c302299f7dae2009fe0053bcd062009" exitCode=0 Jul 29 07:20:59.589018 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.588980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" event={"ID":"9d2d8b8b-30db-472a-9d44-2b9b54d44320","Type":"ContainerDied","Data":"89be277fe52c79f3e3c6f4c2ade6ad1d8c302299f7dae2009fe0053bcd062009"} Jul 29 07:20:59.591214 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.591194 2565 generic.go:356] "Generic (PLEG): container finished" podID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerID="0b6d9251cca1603120b55ef834667f0748346d4a7a674642cb1ce008be387c0f" exitCode=0 Jul 29 07:20:59.591315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.591280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" event={"ID":"c496b17e-3fe8-4094-916f-4ddfb88812f9","Type":"ContainerDied","Data":"0b6d9251cca1603120b55ef834667f0748346d4a7a674642cb1ce008be387c0f"} Jul 29 07:20:59.593456 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.593417 2565 generic.go:356] "Generic (PLEG): container finished" podID="35ece30b-f78d-4175-83df-e665307b551c" containerID="84e7d744b26c56f652ecf0a89d0d672254093e93feecc0845898c2353be986a0" exitCode=0 Jul 29 07:20:59.593538 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:20:59.593499 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" event={"ID":"35ece30b-f78d-4175-83df-e665307b551c","Type":"ContainerDied","Data":"84e7d744b26c56f652ecf0a89d0d672254093e93feecc0845898c2353be986a0"} Jul 29 07:21:00.600400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.600363 2565 generic.go:356] "Generic (PLEG): container finished" podID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerID="a38413937c07962585bfbd40f3e3dd26937bc06d4aaff1f75d1f6c3d839981ac" exitCode=0 Jul 29 07:21:00.600781 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.600456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" event={"ID":"9d2d8b8b-30db-472a-9d44-2b9b54d44320","Type":"ContainerDied","Data":"a38413937c07962585bfbd40f3e3dd26937bc06d4aaff1f75d1f6c3d839981ac"} Jul 29 07:21:00.737311 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.737286 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:21:00.759010 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.758192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-util\") pod \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " Jul 29 07:21:00.759010 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.758236 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-bundle\") pod \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " Jul 29 07:21:00.759010 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.758313 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgxdd\" (UniqueName: \"kubernetes.io/projected/aa36ba9a-0a57-4151-9e05-891e03cd76eb-kube-api-access-mgxdd\") pod \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\" (UID: \"aa36ba9a-0a57-4151-9e05-891e03cd76eb\") " Jul 29 07:21:00.759648 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.759609 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-bundle" (OuterVolumeSpecName: "bundle") pod "aa36ba9a-0a57-4151-9e05-891e03cd76eb" (UID: "aa36ba9a-0a57-4151-9e05-891e03cd76eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:00.762009 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.761983 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa36ba9a-0a57-4151-9e05-891e03cd76eb-kube-api-access-mgxdd" (OuterVolumeSpecName: "kube-api-access-mgxdd") pod "aa36ba9a-0a57-4151-9e05-891e03cd76eb" (UID: "aa36ba9a-0a57-4151-9e05-891e03cd76eb"). InnerVolumeSpecName "kube-api-access-mgxdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:00.765887 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.765861 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-util" (OuterVolumeSpecName: "util") pod "aa36ba9a-0a57-4151-9e05-891e03cd76eb" (UID: "aa36ba9a-0a57-4151-9e05-891e03cd76eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:00.793392 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.793374 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:21:00.796083 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.796065 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:21:00.859096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859032 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-bundle\") pod \"35ece30b-f78d-4175-83df-e665307b551c\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " Jul 29 07:21:00.859096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859060 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-bundle\") pod \"c496b17e-3fe8-4094-916f-4ddfb88812f9\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " Jul 29 07:21:00.859096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859081 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqn8r\" (UniqueName: \"kubernetes.io/projected/c496b17e-3fe8-4094-916f-4ddfb88812f9-kube-api-access-rqn8r\") pod \"c496b17e-3fe8-4094-916f-4ddfb88812f9\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " Jul 29 07:21:00.859315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dld6w\" (UniqueName: \"kubernetes.io/projected/35ece30b-f78d-4175-83df-e665307b551c-kube-api-access-dld6w\") pod \"35ece30b-f78d-4175-83df-e665307b551c\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " Jul 29 07:21:00.859315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859146 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-util\") pod \"c496b17e-3fe8-4094-916f-4ddfb88812f9\" (UID: \"c496b17e-3fe8-4094-916f-4ddfb88812f9\") " Jul 29 07:21:00.859315 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859210 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-util\") pod \"35ece30b-f78d-4175-83df-e665307b551c\" (UID: \"35ece30b-f78d-4175-83df-e665307b551c\") " Jul 29 07:21:00.859573 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859554 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxdd\" (UniqueName: \"kubernetes.io/projected/aa36ba9a-0a57-4151-9e05-891e03cd76eb-kube-api-access-mgxdd\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.859854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859577 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.859854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859592 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa36ba9a-0a57-4151-9e05-891e03cd76eb-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.859854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859655 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-bundle" (OuterVolumeSpecName: "bundle") pod "35ece30b-f78d-4175-83df-e665307b551c" (UID: "35ece30b-f78d-4175-83df-e665307b551c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:00.859854 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.859812 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-bundle" (OuterVolumeSpecName: "bundle") pod "c496b17e-3fe8-4094-916f-4ddfb88812f9" (UID: "c496b17e-3fe8-4094-916f-4ddfb88812f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:00.861184 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.861164 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c496b17e-3fe8-4094-916f-4ddfb88812f9-kube-api-access-rqn8r" (OuterVolumeSpecName: "kube-api-access-rqn8r") pod "c496b17e-3fe8-4094-916f-4ddfb88812f9" (UID: "c496b17e-3fe8-4094-916f-4ddfb88812f9"). InnerVolumeSpecName "kube-api-access-rqn8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:00.861275 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.861257 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ece30b-f78d-4175-83df-e665307b551c-kube-api-access-dld6w" (OuterVolumeSpecName: "kube-api-access-dld6w") pod "35ece30b-f78d-4175-83df-e665307b551c" (UID: "35ece30b-f78d-4175-83df-e665307b551c"). InnerVolumeSpecName "kube-api-access-dld6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:00.864434 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.864397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-util" (OuterVolumeSpecName: "util") pod "35ece30b-f78d-4175-83df-e665307b551c" (UID: "35ece30b-f78d-4175-83df-e665307b551c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:00.864867 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.864846 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-util" (OuterVolumeSpecName: "util") pod "c496b17e-3fe8-4094-916f-4ddfb88812f9" (UID: "c496b17e-3fe8-4094-916f-4ddfb88812f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:00.960257 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.960232 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.960257 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.960251 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.960257 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.960260 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqn8r\" (UniqueName: \"kubernetes.io/projected/c496b17e-3fe8-4094-916f-4ddfb88812f9-kube-api-access-rqn8r\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.960412 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.960270 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dld6w\" (UniqueName: \"kubernetes.io/projected/35ece30b-f78d-4175-83df-e665307b551c-kube-api-access-dld6w\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.960412 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.960278 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c496b17e-3fe8-4094-916f-4ddfb88812f9-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:00.960412 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:00.960285 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35ece30b-f78d-4175-83df-e665307b551c-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:01.606480 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.606443 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" Jul 29 07:21:01.606871 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.606443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb" event={"ID":"c496b17e-3fe8-4094-916f-4ddfb88812f9","Type":"ContainerDied","Data":"aad3b46bf05e1fbfb6996ea48f0b4d9382f8c70bf31982581f40dcd4ef6124f9"} Jul 29 07:21:01.606871 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.606598 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad3b46bf05e1fbfb6996ea48f0b4d9382f8c70bf31982581f40dcd4ef6124f9" Jul 29 07:21:01.608458 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.608427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" event={"ID":"35ece30b-f78d-4175-83df-e665307b551c","Type":"ContainerDied","Data":"ec76a506c0ee4e54db44b19fff91d5c724b47839999a55ad84570b9b0edf4c0c"} Jul 29 07:21:01.608598 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.608478 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec76a506c0ee4e54db44b19fff91d5c724b47839999a55ad84570b9b0edf4c0c" Jul 29 07:21:01.608598 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.608436 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r" Jul 29 07:21:01.610562 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.610536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" event={"ID":"aa36ba9a-0a57-4151-9e05-891e03cd76eb","Type":"ContainerDied","Data":"1d29f56e7120dcfd15d86f867aafec4b52193eaea0a9931cbe7a66c32ac8692e"} Jul 29 07:21:01.610642 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.610581 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d29f56e7120dcfd15d86f867aafec4b52193eaea0a9931cbe7a66c32ac8692e" Jul 29 07:21:01.611150 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.610803 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2" Jul 29 07:21:01.734793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.734770 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:21:01.766708 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.766685 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmjx\" (UniqueName: \"kubernetes.io/projected/9d2d8b8b-30db-472a-9d44-2b9b54d44320-kube-api-access-jgmjx\") pod \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " Jul 29 07:21:01.766813 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.766717 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-bundle\") pod \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " Jul 29 07:21:01.766813 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.766793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-util\") pod \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\" (UID: \"9d2d8b8b-30db-472a-9d44-2b9b54d44320\") " Jul 29 07:21:01.767347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.767316 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-bundle" (OuterVolumeSpecName: "bundle") pod "9d2d8b8b-30db-472a-9d44-2b9b54d44320" (UID: "9d2d8b8b-30db-472a-9d44-2b9b54d44320"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:01.768622 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.768596 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2d8b8b-30db-472a-9d44-2b9b54d44320-kube-api-access-jgmjx" (OuterVolumeSpecName: "kube-api-access-jgmjx") pod "9d2d8b8b-30db-472a-9d44-2b9b54d44320" (UID: "9d2d8b8b-30db-472a-9d44-2b9b54d44320"). InnerVolumeSpecName "kube-api-access-jgmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:01.771514 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.771490 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-util" (OuterVolumeSpecName: "util") pod "9d2d8b8b-30db-472a-9d44-2b9b54d44320" (UID: "9d2d8b8b-30db-472a-9d44-2b9b54d44320"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:01.867921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.867856 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmjx\" (UniqueName: \"kubernetes.io/projected/9d2d8b8b-30db-472a-9d44-2b9b54d44320-kube-api-access-jgmjx\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:01.867921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.867882 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:01.867921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:01.867892 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d2d8b8b-30db-472a-9d44-2b9b54d44320-util\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:02.617171 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:02.617138 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" Jul 29 07:21:02.617544 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:02.617165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb" event={"ID":"9d2d8b8b-30db-472a-9d44-2b9b54d44320","Type":"ContainerDied","Data":"764ac8197268149af226f9a88075b91f067a939ef6d308b017dccd92e698f764"} Jul 29 07:21:02.617544 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:02.617211 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764ac8197268149af226f9a88075b91f067a939ef6d308b017dccd92e698f764" Jul 29 07:21:35.814811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.814771 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c47498bb6-rmxdn"] Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815243 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="extract" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815264 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="extract" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815275 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="extract" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815283 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="extract" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815294 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="util" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815303 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="util" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815313 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="pull" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815321 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="pull" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815341 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="extract" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815350 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="extract" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815366 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="util" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815374 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="util" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815387 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="util" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815394 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="util" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815401 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="pull" Jul 29 07:21:35.815410 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815410 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="pull" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815424 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="util" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815432 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="util" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815440 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="pull" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815448 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="pull" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815458 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="pull" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815466 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="pull" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815478 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="extract" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815486 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="extract" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815574 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa36ba9a-0a57-4151-9e05-891e03cd76eb" containerName="extract" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815591 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c496b17e-3fe8-4094-916f-4ddfb88812f9" containerName="extract" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815602 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2d8b8b-30db-472a-9d44-2b9b54d44320" containerName="extract" Jul 29 07:21:35.816177 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.815612 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ece30b-f78d-4175-83df-e665307b551c" containerName="extract" Jul 29 07:21:35.820320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.820291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.829706 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.829681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c47498bb6-rmxdn"] Jul 29 07:21:35.936120 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936095 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-serving-cert\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.936239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-config\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.936239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-oauth-serving-cert\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.936239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bqg\" (UniqueName: \"kubernetes.io/projected/8eb685d3-7d87-4628-8564-a4bd967d1b7c-kube-api-access-t8bqg\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.936239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-trusted-ca-bundle\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.936375 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-oauth-config\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:35.936375 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:35.936299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-service-ca\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-serving-cert\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037635 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-config\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037706 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-oauth-serving-cert\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037706 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bqg\" (UniqueName: \"kubernetes.io/projected/8eb685d3-7d87-4628-8564-a4bd967d1b7c-kube-api-access-t8bqg\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-trusted-ca-bundle\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-oauth-config\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.037811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.037778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-service-ca\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.038301 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.038279 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-config\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.038405 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.038338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-oauth-serving-cert\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.038537 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.038510 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-service-ca\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.038537 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.038535 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8eb685d3-7d87-4628-8564-a4bd967d1b7c-trusted-ca-bundle\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.040032 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.040008 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-serving-cert\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.040176 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.040156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8eb685d3-7d87-4628-8564-a4bd967d1b7c-console-oauth-config\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.046074 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.046058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bqg\" (UniqueName: \"kubernetes.io/projected/8eb685d3-7d87-4628-8564-a4bd967d1b7c-kube-api-access-t8bqg\") pod \"console-c47498bb6-rmxdn\" (UID: \"8eb685d3-7d87-4628-8564-a4bd967d1b7c\") " pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.129355 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.129296 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:36.258242 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.258219 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c47498bb6-rmxdn"] Jul 29 07:21:36.260324 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:21:36.260293 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb685d3_7d87_4628_8564_a4bd967d1b7c.slice/crio-41097d9a686a1bcfbc29c696c921e3de14964d31b17b3040d6117157001bfd1c WatchSource:0}: Error finding container 41097d9a686a1bcfbc29c696c921e3de14964d31b17b3040d6117157001bfd1c: Status 404 returned error can't find the container with id 41097d9a686a1bcfbc29c696c921e3de14964d31b17b3040d6117157001bfd1c Jul 29 07:21:36.775566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.775525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c47498bb6-rmxdn" event={"ID":"8eb685d3-7d87-4628-8564-a4bd967d1b7c","Type":"ContainerStarted","Data":"ef5347263950ff5a04d9c3615e1ea2313ed67238b4ff1685360d14132059f7e1"} Jul 29 07:21:36.775761 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.775579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c47498bb6-rmxdn" event={"ID":"8eb685d3-7d87-4628-8564-a4bd967d1b7c","Type":"ContainerStarted","Data":"41097d9a686a1bcfbc29c696c921e3de14964d31b17b3040d6117157001bfd1c"} Jul 29 07:21:36.795923 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:36.795869 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c47498bb6-rmxdn" podStartSLOduration=1.7958536550000002 podStartE2EDuration="1.795853655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:36.794895136 +0000 UTC m=+378.401073294" watchObservedRunningTime="2026-07-29 07:21:36.795853655 +0000 UTC m=+378.402031762" Jul 29 07:21:45.116399 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.116369 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5"] Jul 29 07:21:45.119529 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.119504 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.122987 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.122946 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 07:21:45.123140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.122988 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 07:21:45.123140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.123045 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nbw6s\"" Jul 29 07:21:45.136826 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.136804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5"] Jul 29 07:21:45.207540 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.207509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe289e0-e2f0-4727-aabc-dcfbe5954767-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.207680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.207548 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c88z\" (UniqueName: \"kubernetes.io/projected/fbe289e0-e2f0-4727-aabc-dcfbe5954767-kube-api-access-9c88z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.308363 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.308332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe289e0-e2f0-4727-aabc-dcfbe5954767-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.308363 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.308362 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c88z\" (UniqueName: \"kubernetes.io/projected/fbe289e0-e2f0-4727-aabc-dcfbe5954767-kube-api-access-9c88z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.308697 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.308679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe289e0-e2f0-4727-aabc-dcfbe5954767-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.320292 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.320268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c88z\" (UniqueName: \"kubernetes.io/projected/fbe289e0-e2f0-4727-aabc-dcfbe5954767-kube-api-access-9c88z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.430719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.430660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:45.567429 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.567390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5"] Jul 29 07:21:45.573526 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:21:45.573499 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe289e0_e2f0_4727_aabc_dcfbe5954767.slice/crio-efe02956de0aadeedda48716854fd8f79eef3b5cd036a984c151bdf4d4bafef0 WatchSource:0}: Error finding container efe02956de0aadeedda48716854fd8f79eef3b5cd036a984c151bdf4d4bafef0: Status 404 returned error can't find the container with id efe02956de0aadeedda48716854fd8f79eef3b5cd036a984c151bdf4d4bafef0 Jul 29 07:21:45.760198 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.760116 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5"] Jul 29 07:21:45.768738 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.768711 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5"] Jul 29 07:21:45.783018 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.782989 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj"] Jul 29 07:21:45.787261 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.787240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:45.805616 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.805589 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj"] Jul 29 07:21:45.913995 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.913939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8216a10f-52da-4c63-8a79-e67afef61326-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hqgbj\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:45.914159 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:45.914063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nstc\" (UniqueName: \"kubernetes.io/projected/8216a10f-52da-4c63-8a79-e67afef61326-kube-api-access-2nstc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hqgbj\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:46.015233 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.015155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nstc\" (UniqueName: \"kubernetes.io/projected/8216a10f-52da-4c63-8a79-e67afef61326-kube-api-access-2nstc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hqgbj\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:46.015367 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.015300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8216a10f-52da-4c63-8a79-e67afef61326-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hqgbj\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:46.015636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.015607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8216a10f-52da-4c63-8a79-e67afef61326-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hqgbj\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:46.025202 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.025177 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nstc\" (UniqueName: \"kubernetes.io/projected/8216a10f-52da-4c63-8a79-e67afef61326-kube-api-access-2nstc\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hqgbj\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:46.098355 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.098329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:46.130209 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.130172 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:46.130636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.130220 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:46.136222 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.136196 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:46.273347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.273315 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj"] Jul 29 07:21:46.312924 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:21:46.312893 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8216a10f_52da_4c63_8a79_e67afef61326.slice/crio-81fef169676fae8ac844296ef04d6f9b2fa016ab40160a9075ea300a9631f5cd WatchSource:0}: Error finding container 81fef169676fae8ac844296ef04d6f9b2fa016ab40160a9075ea300a9631f5cd: Status 404 returned error can't find the container with id 81fef169676fae8ac844296ef04d6f9b2fa016ab40160a9075ea300a9631f5cd Jul 29 07:21:46.832546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.832477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" event={"ID":"8216a10f-52da-4c63-8a79-e67afef61326","Type":"ContainerStarted","Data":"81fef169676fae8ac844296ef04d6f9b2fa016ab40160a9075ea300a9631f5cd"} Jul 29 07:21:46.837241 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.837215 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c47498bb6-rmxdn" Jul 29 07:21:46.894217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:46.894183 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c697c4754-4hsrk"] Jul 29 07:21:50.855913 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:50.855868 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" containerName="manager" containerID="cri-o://87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96" gracePeriod=2 Jul 29 07:21:50.857568 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:50.857537 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" event={"ID":"8216a10f-52da-4c63-8a79-e67afef61326","Type":"ContainerStarted","Data":"949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12"} Jul 29 07:21:50.857702 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:50.857596 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:21:50.858341 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:21:50.858307 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:50.860598 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:21:50.860547 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:50.880926 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:50.880888 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" podStartSLOduration=2.283871634 podStartE2EDuration="5.88087254s" totalImagesPullingTime="3.597000906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:21:50.879188748 +0000 UTC m=+392.485366884" watchObservedRunningTime="2026-07-29 07:21:50.88087254 +0000 UTC m=+392.487050650" Jul 29 07:21:51.095040 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.095007 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:51.097549 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:21:51.097514 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:51.262146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.262074 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe289e0-e2f0-4727-aabc-dcfbe5954767-extensions-socket-volume\") pod \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " Jul 29 07:21:51.262295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.262148 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c88z\" (UniqueName: \"kubernetes.io/projected/fbe289e0-e2f0-4727-aabc-dcfbe5954767-kube-api-access-9c88z\") pod \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\" (UID: \"fbe289e0-e2f0-4727-aabc-dcfbe5954767\") " Jul 29 07:21:51.262355 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.262306 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe289e0-e2f0-4727-aabc-dcfbe5954767-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fbe289e0-e2f0-4727-aabc-dcfbe5954767" (UID: "fbe289e0-e2f0-4727-aabc-dcfbe5954767"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:21:51.264052 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.264031 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe289e0-e2f0-4727-aabc-dcfbe5954767-kube-api-access-9c88z" (OuterVolumeSpecName: "kube-api-access-9c88z") pod "fbe289e0-e2f0-4727-aabc-dcfbe5954767" (UID: "fbe289e0-e2f0-4727-aabc-dcfbe5954767"). InnerVolumeSpecName "kube-api-access-9c88z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:21:51.363328 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.363304 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe289e0-e2f0-4727-aabc-dcfbe5954767-extensions-socket-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:51.363328 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.363325 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c88z\" (UniqueName: \"kubernetes.io/projected/fbe289e0-e2f0-4727-aabc-dcfbe5954767-kube-api-access-9c88z\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:21:51.864040 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.864007 2565 generic.go:356] "Generic (PLEG): container finished" podID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" containerID="87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96" exitCode=2 Jul 29 07:21:51.864440 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.864065 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:51.864440 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.864109 2565 scope.go:117] "RemoveContainer" containerID="87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96" Jul 29 07:21:51.866712 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:21:51.866686 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:51.873128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.873112 2565 scope.go:117] "RemoveContainer" containerID="87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96" Jul 29 07:21:51.873369 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:21:51.873354 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96\": container with ID starting with 87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96 not found: ID does not exist" containerID="87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96" Jul 29 07:21:51.873405 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:51.873376 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96"} err="failed to get container status \"87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96\": rpc error: code = NotFound desc = could not find container \"87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96\": container with ID starting with 87aeff3d3a52d73d694a400ca260c3b0a739b2657f04bb3a6278c1610d081b96 not found: ID does not exist" Jul 29 07:21:51.874793 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:21:51.874772 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tfbz5\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tfbz5" Jul 29 07:21:53.019277 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:21:53.019242 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" path="/var/lib/kubelet/pods/fbe289e0-e2f0-4727-aabc-dcfbe5954767/volumes" Jul 29 07:22:01.866785 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:01.866751 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:22:11.918842 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:11.918780 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c697c4754-4hsrk" podUID="7f780598-0017-4302-a8a7-2d042f864fbd" containerName="console" containerID="cri-o://f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf" gracePeriod=15 Jul 29 07:22:12.164686 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.164662 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:22:12.235865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.235801 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvd8r\" (UniqueName: \"kubernetes.io/projected/7f780598-0017-4302-a8a7-2d042f864fbd-kube-api-access-wvd8r\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.235865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.235855 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-oauth-serving-cert\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.236054 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.235875 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-service-ca\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.236054 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.235924 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-oauth-config\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.236054 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.235940 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-serving-cert\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.236054 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.235983 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-console-config\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.236054 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.236025 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-trusted-ca-bundle\") pod \"7f780598-0017-4302-a8a7-2d042f864fbd\" (UID: \"7f780598-0017-4302-a8a7-2d042f864fbd\") " Jul 29 07:22:12.236369 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.236341 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:12.236436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.236398 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-console-config" (OuterVolumeSpecName: "console-config") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:12.236436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.236421 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:12.236652 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.236629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:22:12.238046 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.238020 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:22:12.238386 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.238368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:22:12.238386 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.238373 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f780598-0017-4302-a8a7-2d042f864fbd-kube-api-access-wvd8r" (OuterVolumeSpecName: "kube-api-access-wvd8r") pod "7f780598-0017-4302-a8a7-2d042f864fbd" (UID: "7f780598-0017-4302-a8a7-2d042f864fbd"). InnerVolumeSpecName "kube-api-access-wvd8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:12.337327 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337297 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-oauth-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.337327 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337325 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f780598-0017-4302-a8a7-2d042f864fbd-console-serving-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.337488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337340 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-console-config\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.337488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337352 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-trusted-ca-bundle\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.337488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337363 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvd8r\" (UniqueName: \"kubernetes.io/projected/7f780598-0017-4302-a8a7-2d042f864fbd-kube-api-access-wvd8r\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.337488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337375 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-oauth-serving-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.337488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.337387 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f780598-0017-4302-a8a7-2d042f864fbd-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:12.967802 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.967767 2565 generic.go:356] "Generic (PLEG): container finished" podID="7f780598-0017-4302-a8a7-2d042f864fbd" containerID="f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf" exitCode=0 Jul 29 07:22:12.968212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.967817 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c697c4754-4hsrk" Jul 29 07:22:12.968212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.967861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c697c4754-4hsrk" event={"ID":"7f780598-0017-4302-a8a7-2d042f864fbd","Type":"ContainerDied","Data":"f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf"} Jul 29 07:22:12.968212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.967914 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c697c4754-4hsrk" event={"ID":"7f780598-0017-4302-a8a7-2d042f864fbd","Type":"ContainerDied","Data":"cf43a09ba89ebbc02301e4ee8b9a2ef447557d6a52776a4fc5127eb93f076ab7"} Jul 29 07:22:12.968212 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.967935 2565 scope.go:117] "RemoveContainer" containerID="f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf" Jul 29 07:22:12.976789 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.976772 2565 scope.go:117] "RemoveContainer" containerID="f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf" Jul 29 07:22:12.977046 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:22:12.977027 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf\": container with ID starting with f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf not found: ID does not exist" containerID="f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf" Jul 29 07:22:12.977102 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.977054 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf"} err="failed to get container status \"f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf\": rpc error: code = NotFound desc = could not find container \"f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf\": container with ID starting with f111844e27cd3a9dbced8650678cef4d93aaceab9ee19a9382b736bea7d6cfcf not found: ID does not exist" Jul 29 07:22:12.989899 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.989873 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c697c4754-4hsrk"] Jul 29 07:22:12.994220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:12.994198 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c697c4754-4hsrk"] Jul 29 07:22:13.019412 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:13.019388 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f780598-0017-4302-a8a7-2d042f864fbd" path="/var/lib/kubelet/pods/7f780598-0017-4302-a8a7-2d042f864fbd/volumes" Jul 29 07:22:14.167386 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.167351 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj"] Jul 29 07:22:14.167818 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.167552 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" podUID="8216a10f-52da-4c63-8a79-e67afef61326" containerName="manager" containerID="cri-o://949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12" gracePeriod=10 Jul 29 07:22:14.417663 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.417606 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:22:14.453772 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.453748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8216a10f-52da-4c63-8a79-e67afef61326-extensions-socket-volume\") pod \"8216a10f-52da-4c63-8a79-e67afef61326\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " Jul 29 07:22:14.453897 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.453807 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nstc\" (UniqueName: \"kubernetes.io/projected/8216a10f-52da-4c63-8a79-e67afef61326-kube-api-access-2nstc\") pod \"8216a10f-52da-4c63-8a79-e67afef61326\" (UID: \"8216a10f-52da-4c63-8a79-e67afef61326\") " Jul 29 07:22:14.454226 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.454203 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8216a10f-52da-4c63-8a79-e67afef61326-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8216a10f-52da-4c63-8a79-e67afef61326" (UID: "8216a10f-52da-4c63-8a79-e67afef61326"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:22:14.455737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.455716 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8216a10f-52da-4c63-8a79-e67afef61326-kube-api-access-2nstc" (OuterVolumeSpecName: "kube-api-access-2nstc") pod "8216a10f-52da-4c63-8a79-e67afef61326" (UID: "8216a10f-52da-4c63-8a79-e67afef61326"). InnerVolumeSpecName "kube-api-access-2nstc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:22:14.555095 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.555069 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8216a10f-52da-4c63-8a79-e67afef61326-extensions-socket-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:14.555095 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.555093 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nstc\" (UniqueName: \"kubernetes.io/projected/8216a10f-52da-4c63-8a79-e67afef61326-kube-api-access-2nstc\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:22:14.979895 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.979854 2565 generic.go:356] "Generic (PLEG): container finished" podID="8216a10f-52da-4c63-8a79-e67afef61326" containerID="949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12" exitCode=0 Jul 29 07:22:14.980080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.979902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" event={"ID":"8216a10f-52da-4c63-8a79-e67afef61326","Type":"ContainerDied","Data":"949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12"} Jul 29 07:22:14.980080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.979913 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" Jul 29 07:22:14.980080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.980021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj" event={"ID":"8216a10f-52da-4c63-8a79-e67afef61326","Type":"ContainerDied","Data":"81fef169676fae8ac844296ef04d6f9b2fa016ab40160a9075ea300a9631f5cd"} Jul 29 07:22:14.980080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.980053 2565 scope.go:117] "RemoveContainer" containerID="949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12" Jul 29 07:22:14.989075 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.989056 2565 scope.go:117] "RemoveContainer" containerID="949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12" Jul 29 07:22:14.989308 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:22:14.989286 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12\": container with ID starting with 949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12 not found: ID does not exist" containerID="949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12" Jul 29 07:22:14.989351 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:14.989314 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12"} err="failed to get container status \"949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12\": rpc error: code = NotFound desc = could not find container \"949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12\": container with ID starting with 949c0180f0820f748ce53720f3f55a0af78cd5d0bbf6c37abf30a469bd1d2a12 not found: ID does not exist" Jul 29 07:22:15.019243 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:15.019221 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj"] Jul 29 07:22:15.019351 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:15.019251 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hqgbj"] Jul 29 07:22:17.018304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:17.018270 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8216a10f-52da-4c63-8a79-e67afef61326" path="/var/lib/kubelet/pods/8216a10f-52da-4c63-8a79-e67afef61326/volumes" Jul 29 07:22:18.368595 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368561 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm"] Jul 29 07:22:18.368971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368866 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" containerName="manager" Jul 29 07:22:18.368971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368876 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" containerName="manager" Jul 29 07:22:18.368971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368895 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8216a10f-52da-4c63-8a79-e67afef61326" containerName="manager" Jul 29 07:22:18.368971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368902 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8216a10f-52da-4c63-8a79-e67afef61326" containerName="manager" Jul 29 07:22:18.368971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368940 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f780598-0017-4302-a8a7-2d042f864fbd" containerName="console" Jul 29 07:22:18.368971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.368947 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f780598-0017-4302-a8a7-2d042f864fbd" containerName="console" Jul 29 07:22:18.369198 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.369015 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f780598-0017-4302-a8a7-2d042f864fbd" containerName="console" Jul 29 07:22:18.369198 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.369023 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe289e0-e2f0-4727-aabc-dcfbe5954767" containerName="manager" Jul 29 07:22:18.369198 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.369031 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8216a10f-52da-4c63-8a79-e67afef61326" containerName="manager" Jul 29 07:22:18.371843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.371819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.375285 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.375266 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqwff\"" Jul 29 07:22:18.388639 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.388613 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm"] Jul 29 07:22:18.484883 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.484849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.484883 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.484887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfts\" (UniqueName: \"kubernetes.io/projected/a2fd1ef4-4586-455d-99f0-875fccfcdd72-kube-api-access-cmfts\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.484915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.484989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.485035 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.485060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.485129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.485184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.485220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.485208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfts\" (UniqueName: \"kubernetes.io/projected/a2fd1ef4-4586-455d-99f0-875fccfcdd72-kube-api-access-cmfts\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.587237 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.586783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.588404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.588378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.588886 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.588609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.589034 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.588823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.589114 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.589082 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.589424 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.589406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.592136 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.592116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.594312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.594291 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.601830 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.601809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fd1ef4-4586-455d-99f0-875fccfcdd72-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.602065 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.602041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfts\" (UniqueName: \"kubernetes.io/projected/a2fd1ef4-4586-455d-99f0-875fccfcdd72-kube-api-access-cmfts\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vr8nm\" (UID: \"a2fd1ef4-4586-455d-99f0-875fccfcdd72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.682096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.682029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:18.821592 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.821567 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm"] Jul 29 07:22:18.823169 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:22:18.823148 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fd1ef4_4586_455d_99f0_875fccfcdd72.slice/crio-9cdd80162131a7fc73e121a7bf54636a8250effdcaea128336505ac04da7e7ab WatchSource:0}: Error finding container 9cdd80162131a7fc73e121a7bf54636a8250effdcaea128336505ac04da7e7ab: Status 404 returned error can't find the container with id 9cdd80162131a7fc73e121a7bf54636a8250effdcaea128336505ac04da7e7ab Jul 29 07:22:18.825777 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.825748 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:22:18.825835 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.825808 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:22:18.825867 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:18.825836 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:22:19.001879 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:19.001787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" event={"ID":"a2fd1ef4-4586-455d-99f0-875fccfcdd72","Type":"ContainerStarted","Data":"e375462dd2f428e3694528a099d05698fe511a3dbbbfc65670fc3ed8fdbfb3ac"} Jul 29 07:22:19.002065 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:19.001880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" event={"ID":"a2fd1ef4-4586-455d-99f0-875fccfcdd72","Type":"ContainerStarted","Data":"9cdd80162131a7fc73e121a7bf54636a8250effdcaea128336505ac04da7e7ab"} Jul 29 07:22:19.029905 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:19.029858 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" podStartSLOduration=1.029843384 podStartE2EDuration="1.029843384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:22:19.02958575 +0000 UTC m=+420.635763859" watchObservedRunningTime="2026-07-29 07:22:19.029843384 +0000 UTC m=+420.636021493" Jul 29 07:22:19.682610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:19.682580 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:19.687140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:19.687118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:20.006296 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:20.006222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:22:20.006948 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:22:20.006932 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:23:17.849734 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.849698 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5b4b8496-6fczn"] Jul 29 07:23:17.852133 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.852117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:17.861099 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.861081 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Jul 29 07:23:17.861612 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.861595 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 07:23:17.861699 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.861614 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 07:23:17.865963 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.865940 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qpfx2\"" Jul 29 07:23:17.889463 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.889430 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b4b8496-6fczn"] Jul 29 07:23:17.970483 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.970452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59fq\" (UniqueName: \"kubernetes.io/projected/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-kube-api-access-q59fq\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:17.970600 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.970506 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-webhook-cert\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:17.970600 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:17.970569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-metrics-certs\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.071131 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.071101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59fq\" (UniqueName: \"kubernetes.io/projected/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-kube-api-access-q59fq\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.071278 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.071151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-webhook-cert\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.071278 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.071174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-metrics-certs\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.073482 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.073451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-metrics-certs\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.073603 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.073559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-webhook-cert\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.094038 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.094007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59fq\" (UniqueName: \"kubernetes.io/projected/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-kube-api-access-q59fq\") pod \"maas-controller-6c5b4b8496-6fczn\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.162387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.162330 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:18.337045 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:18.337018 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5b4b8496-6fczn"] Jul 29 07:23:18.338562 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:23:18.338535 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b5dfe4_caa6_44df_ae9a_980a5d5490c1.slice/crio-1a9a0dd3ed842fe0c102bd51f1eeaea4e4cd1f78d28a0baecd558af588c43342 WatchSource:0}: Error finding container 1a9a0dd3ed842fe0c102bd51f1eeaea4e4cd1f78d28a0baecd558af588c43342: Status 404 returned error can't find the container with id 1a9a0dd3ed842fe0c102bd51f1eeaea4e4cd1f78d28a0baecd558af588c43342 Jul 29 07:23:19.292331 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:19.292284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" event={"ID":"14b5dfe4-caa6-44df-ae9a-980a5d5490c1","Type":"ContainerStarted","Data":"1a9a0dd3ed842fe0c102bd51f1eeaea4e4cd1f78d28a0baecd558af588c43342"} Jul 29 07:23:20.711902 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:20.711880 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 07:23:21.304964 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:21.304916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" event={"ID":"14b5dfe4-caa6-44df-ae9a-980a5d5490c1","Type":"ContainerStarted","Data":"02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d"} Jul 29 07:23:21.305120 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:21.305072 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:21.328300 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:21.328255 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" podStartSLOduration=1.959389744 podStartE2EDuration="4.328241015s" totalImagesPullingTime="2.368851271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:21.326729807 +0000 UTC m=+482.932907915" watchObservedRunningTime="2026-07-29 07:23:21.328241015 +0000 UTC m=+482.934419124" Jul 29 07:23:32.311940 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.311909 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:32.619510 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.619434 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5b4b8496-6fczn"] Jul 29 07:23:32.619669 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.619616 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" podUID="14b5dfe4-caa6-44df-ae9a-980a5d5490c1" containerName="manager" containerID="cri-o://02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d" gracePeriod=30 Jul 29 07:23:32.864516 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.864490 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:32.997106 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.997040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q59fq\" (UniqueName: \"kubernetes.io/projected/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-kube-api-access-q59fq\") pod \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " Jul 29 07:23:32.997231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.997115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-webhook-cert\") pod \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " Jul 29 07:23:32.997231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.997193 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-metrics-certs\") pod \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\" (UID: \"14b5dfe4-caa6-44df-ae9a-980a5d5490c1\") " Jul 29 07:23:32.999321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.999290 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-kube-api-access-q59fq" (OuterVolumeSpecName: "kube-api-access-q59fq") pod "14b5dfe4-caa6-44df-ae9a-980a5d5490c1" (UID: "14b5dfe4-caa6-44df-ae9a-980a5d5490c1"). InnerVolumeSpecName "kube-api-access-q59fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:23:32.999430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.999325 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14b5dfe4-caa6-44df-ae9a-980a5d5490c1" (UID: "14b5dfe4-caa6-44df-ae9a-980a5d5490c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:23:32.999430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:32.999369 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "14b5dfe4-caa6-44df-ae9a-980a5d5490c1" (UID: "14b5dfe4-caa6-44df-ae9a-980a5d5490c1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:23:33.098556 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.098528 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-metrics-certs\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:23:33.098556 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.098554 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59fq\" (UniqueName: \"kubernetes.io/projected/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-kube-api-access-q59fq\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:23:33.098696 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.098567 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14b5dfe4-caa6-44df-ae9a-980a5d5490c1-webhook-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:23:33.366006 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.365945 2565 generic.go:356] "Generic (PLEG): container finished" podID="14b5dfe4-caa6-44df-ae9a-980a5d5490c1" containerID="02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d" exitCode=0 Jul 29 07:23:33.366006 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.366005 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" Jul 29 07:23:33.366504 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.366052 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" event={"ID":"14b5dfe4-caa6-44df-ae9a-980a5d5490c1","Type":"ContainerDied","Data":"02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d"} Jul 29 07:23:33.366504 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.366108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5b4b8496-6fczn" event={"ID":"14b5dfe4-caa6-44df-ae9a-980a5d5490c1","Type":"ContainerDied","Data":"1a9a0dd3ed842fe0c102bd51f1eeaea4e4cd1f78d28a0baecd558af588c43342"} Jul 29 07:23:33.366504 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.366141 2565 scope.go:117] "RemoveContainer" containerID="02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d" Jul 29 07:23:33.374753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.374736 2565 scope.go:117] "RemoveContainer" containerID="02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d" Jul 29 07:23:33.375018 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:23:33.374995 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d\": container with ID starting with 02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d not found: ID does not exist" containerID="02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d" Jul 29 07:23:33.375086 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.375025 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d"} err="failed to get container status \"02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d\": rpc error: code = NotFound desc = could not find container \"02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d\": container with ID starting with 02c6e151af8953a745c7e15a68c6b44dbb5a83d156f3a840c82c6204f03b1d8d not found: ID does not exist" Jul 29 07:23:33.386889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.386865 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5b4b8496-6fczn"] Jul 29 07:23:33.393984 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:33.393695 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c5b4b8496-6fczn"] Jul 29 07:23:35.019828 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:35.019799 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b5dfe4-caa6-44df-ae9a-980a5d5490c1" path="/var/lib/kubelet/pods/14b5dfe4-caa6-44df-ae9a-980a5d5490c1/volumes" Jul 29 07:23:54.965264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:54.965228 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb"] Jul 29 07:23:54.965719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:54.965547 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b5dfe4-caa6-44df-ae9a-980a5d5490c1" containerName="manager" Jul 29 07:23:54.965719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:54.965558 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b5dfe4-caa6-44df-ae9a-980a5d5490c1" containerName="manager" Jul 29 07:23:54.965719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:54.965617 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b5dfe4-caa6-44df-ae9a-980a5d5490c1" containerName="manager" Jul 29 07:23:54.998836 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:54.998814 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.007095 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.007069 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 07:23:55.009646 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.009625 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb"] Jul 29 07:23:55.061620 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.061587 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8npf2"] Jul 29 07:23:55.066528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.066512 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.074856 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.074837 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z87pq\"" Jul 29 07:23:55.085284 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.085262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2875f89-c8e0-4337-aee1-f043919045df-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-q6wfb\" (UID: \"d2875f89-c8e0-4337-aee1-f043919045df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.085368 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.085299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjqm\" (UniqueName: \"kubernetes.io/projected/d2875f89-c8e0-4337-aee1-f043919045df-kube-api-access-5zjqm\") pod \"payload-pre-processing-687dbb6cf6-q6wfb\" (UID: \"d2875f89-c8e0-4337-aee1-f043919045df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.092275 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.092252 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8npf2"] Jul 29 07:23:55.186376 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.186344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjqm\" (UniqueName: \"kubernetes.io/projected/d2875f89-c8e0-4337-aee1-f043919045df-kube-api-access-5zjqm\") pod \"payload-pre-processing-687dbb6cf6-q6wfb\" (UID: \"d2875f89-c8e0-4337-aee1-f043919045df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.186530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.186416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gt8\" (UniqueName: \"kubernetes.io/projected/855fe327-9db6-4515-a618-2ee91f801bc8-kube-api-access-s4gt8\") pod \"payload-processing-5cc6cd444b-8npf2\" (UID: \"855fe327-9db6-4515-a618-2ee91f801bc8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.186530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.186464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2875f89-c8e0-4337-aee1-f043919045df-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-q6wfb\" (UID: \"d2875f89-c8e0-4337-aee1-f043919045df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.186530 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.186488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855fe327-9db6-4515-a618-2ee91f801bc8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8npf2\" (UID: \"855fe327-9db6-4515-a618-2ee91f801bc8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.187203 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.187183 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2875f89-c8e0-4337-aee1-f043919045df-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-q6wfb\" (UID: \"d2875f89-c8e0-4337-aee1-f043919045df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.224492 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.224423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjqm\" (UniqueName: \"kubernetes.io/projected/d2875f89-c8e0-4337-aee1-f043919045df-kube-api-access-5zjqm\") pod \"payload-pre-processing-687dbb6cf6-q6wfb\" (UID: \"d2875f89-c8e0-4337-aee1-f043919045df\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.287885 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.287852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gt8\" (UniqueName: \"kubernetes.io/projected/855fe327-9db6-4515-a618-2ee91f801bc8-kube-api-access-s4gt8\") pod \"payload-processing-5cc6cd444b-8npf2\" (UID: \"855fe327-9db6-4515-a618-2ee91f801bc8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.288063 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.287904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855fe327-9db6-4515-a618-2ee91f801bc8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8npf2\" (UID: \"855fe327-9db6-4515-a618-2ee91f801bc8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.288566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.288545 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855fe327-9db6-4515-a618-2ee91f801bc8-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8npf2\" (UID: \"855fe327-9db6-4515-a618-2ee91f801bc8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.309468 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.309443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:55.323129 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.323108 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gt8\" (UniqueName: \"kubernetes.io/projected/855fe327-9db6-4515-a618-2ee91f801bc8-kube-api-access-s4gt8\") pod \"payload-processing-5cc6cd444b-8npf2\" (UID: \"855fe327-9db6-4515-a618-2ee91f801bc8\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.376462 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.376200 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:55.479667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.479598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb"] Jul 29 07:23:55.480210 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:23:55.480187 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2875f89_c8e0_4337_aee1_f043919045df.slice/crio-3fa128215bc6b8e1693f1ccf45980c8d51a1c7e9db93459d03b7b1e25c4203c1 WatchSource:0}: Error finding container 3fa128215bc6b8e1693f1ccf45980c8d51a1c7e9db93459d03b7b1e25c4203c1: Status 404 returned error can't find the container with id 3fa128215bc6b8e1693f1ccf45980c8d51a1c7e9db93459d03b7b1e25c4203c1 Jul 29 07:23:55.541864 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:23:55.541832 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855fe327_9db6_4515_a618_2ee91f801bc8.slice/crio-0ec853bdef92a94efd555007109ac4790475d3aa13eb49f8f25665c909ce5f14 WatchSource:0}: Error finding container 0ec853bdef92a94efd555007109ac4790475d3aa13eb49f8f25665c909ce5f14: Status 404 returned error can't find the container with id 0ec853bdef92a94efd555007109ac4790475d3aa13eb49f8f25665c909ce5f14 Jul 29 07:23:55.542107 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:55.542089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8npf2"] Jul 29 07:23:56.483889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:56.483838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" event={"ID":"855fe327-9db6-4515-a618-2ee91f801bc8","Type":"ContainerStarted","Data":"0ec853bdef92a94efd555007109ac4790475d3aa13eb49f8f25665c909ce5f14"} Jul 29 07:23:56.487172 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:56.487138 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" event={"ID":"d2875f89-c8e0-4337-aee1-f043919045df","Type":"ContainerStarted","Data":"3fa128215bc6b8e1693f1ccf45980c8d51a1c7e9db93459d03b7b1e25c4203c1"} Jul 29 07:23:58.499682 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:58.499621 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" event={"ID":"855fe327-9db6-4515-a618-2ee91f801bc8","Type":"ContainerStarted","Data":"b31dba572c0d2e3b29ec0ff92e5a8c7f8a393e5509d37153851b196d2d5e6ce0"} Jul 29 07:23:58.500170 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:58.499708 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:23:58.501438 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:58.501412 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" event={"ID":"d2875f89-c8e0-4337-aee1-f043919045df","Type":"ContainerStarted","Data":"8b5c9920ad13322b383573d43a300c22082b7a93da046784b956c4f920958a97"} Jul 29 07:23:58.501549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:58.501492 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:23:58.576817 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:58.576755 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" podStartSLOduration=1.431426479 podStartE2EDuration="3.576732648s" totalImagesPullingTime="2.145306169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:58.533436649 +0000 UTC m=+520.139614757" watchObservedRunningTime="2026-07-29 07:23:58.576732648 +0000 UTC m=+520.182910756" Jul 29 07:23:58.578320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:23:58.578279 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" podStartSLOduration=2.381373885 podStartE2EDuration="4.578265669s" totalImagesPullingTime="2.196891784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:58.577143634 +0000 UTC m=+520.183321738" watchObservedRunningTime="2026-07-29 07:23:58.578265669 +0000 UTC m=+520.184443776" Jul 29 07:24:03.530066 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:03.530031 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2fd1ef4-4586-455d-99f0-875fccfcdd72" containerID="e375462dd2f428e3694528a099d05698fe511a3dbbbfc65670fc3ed8fdbfb3ac" exitCode=0 Jul 29 07:24:03.530604 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:03.530105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" event={"ID":"a2fd1ef4-4586-455d-99f0-875fccfcdd72","Type":"ContainerDied","Data":"e375462dd2f428e3694528a099d05698fe511a3dbbbfc65670fc3ed8fdbfb3ac"} Jul 29 07:24:03.530604 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:03.530571 2565 scope.go:117] "RemoveContainer" containerID="e375462dd2f428e3694528a099d05698fe511a3dbbbfc65670fc3ed8fdbfb3ac" Jul 29 07:24:03.531587 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:03.531561 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:24:03.531662 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:03.531628 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:24:03.531698 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:03.531659 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:24:04.538685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:04.538653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" event={"ID":"a2fd1ef4-4586-455d-99f0-875fccfcdd72","Type":"ContainerStarted","Data":"a60e317d94f83f160e206809485931cf8cfcfbbc08e81dba550145e82d10a5df"} Jul 29 07:24:04.682799 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:04.682764 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:24:04.683901 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:04.683869 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vr8nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 29 07:24:04.684069 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:04.683926 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" podUID="a2fd1ef4-4586-455d-99f0-875fccfcdd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 29 07:24:05.006811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:05.006773 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:24:05.682470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:05.682433 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vr8nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 29 07:24:05.682852 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:05.682490 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" podUID="a2fd1ef4-4586-455d-99f0-875fccfcdd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 29 07:24:06.683016 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:06.682981 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vr8nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 29 07:24:06.683453 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:06.683039 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" podUID="a2fd1ef4-4586-455d-99f0-875fccfcdd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 29 07:24:07.682701 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:07.682666 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vr8nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 29 07:24:07.682880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:07.682724 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" podUID="a2fd1ef4-4586-455d-99f0-875fccfcdd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 29 07:24:08.686196 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:08.686165 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:24:08.687364 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:08.687342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vr8nm" Jul 29 07:24:09.507782 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:09.507748 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8npf2" Jul 29 07:24:09.507991 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:09.507909 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-q6wfb" Jul 29 07:24:10.060239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.060206 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw"] Jul 29 07:24:10.062610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.062592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.068602 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.068581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4s7sl\"" Jul 29 07:24:10.068733 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.068629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 29 07:24:10.068856 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.068836 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 07:24:10.068939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.068871 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 07:24:10.097762 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.097740 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw"] Jul 29 07:24:10.119758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.119731 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aa9a41-1ca6-4798-b278-577bf7249aed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.119855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.119759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.119855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.119835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.119855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.119851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.119980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.119879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.119980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.119912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlszk\" (UniqueName: \"kubernetes.io/projected/16aa9a41-1ca6-4798-b278-577bf7249aed-kube-api-access-nlszk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221240 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221240 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221471 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221471 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlszk\" (UniqueName: \"kubernetes.io/projected/16aa9a41-1ca6-4798-b278-577bf7249aed-kube-api-access-nlszk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221471 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aa9a41-1ca6-4798-b278-577bf7249aed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221763 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.221939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.221858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.223517 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.223497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aa9a41-1ca6-4798-b278-577bf7249aed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.223728 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.223711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aa9a41-1ca6-4798-b278-577bf7249aed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.233016 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.232995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlszk\" (UniqueName: \"kubernetes.io/projected/16aa9a41-1ca6-4798-b278-577bf7249aed-kube-api-access-nlszk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw\" (UID: \"16aa9a41-1ca6-4798-b278-577bf7249aed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.373171 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.373068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:10.516601 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.516571 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw"] Jul 29 07:24:10.516987 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:24:10.516947 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16aa9a41_1ca6_4798_b278_577bf7249aed.slice/crio-16a16124dcf2890ed136f7867d9d9606fce444f4c8846dab3d2b2fe57cb0db9e WatchSource:0}: Error finding container 16a16124dcf2890ed136f7867d9d9606fce444f4c8846dab3d2b2fe57cb0db9e: Status 404 returned error can't find the container with id 16a16124dcf2890ed136f7867d9d9606fce444f4c8846dab3d2b2fe57cb0db9e Jul 29 07:24:10.580014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:10.579979 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" event={"ID":"16aa9a41-1ca6-4798-b278-577bf7249aed","Type":"ContainerStarted","Data":"16a16124dcf2890ed136f7867d9d9606fce444f4c8846dab3d2b2fe57cb0db9e"} Jul 29 07:24:12.591674 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:12.591638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" event={"ID":"16aa9a41-1ca6-4798-b278-577bf7249aed","Type":"ContainerStarted","Data":"2e994d608d11d64f37f1d194def9c1c2f0faccaec0790e40b3659cc16fd9c4e1"} Jul 29 07:24:12.658941 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:12.658892 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" podStartSLOduration=1.601745778 podStartE2EDuration="2.658875013s" totalImagesPullingTime="1.057129235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:24:12.655701847 +0000 UTC m=+534.261879957" watchObservedRunningTime="2026-07-29 07:24:12.658875013 +0000 UTC m=+534.265053150" Jul 29 07:24:20.374000 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:20.373944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:20.374386 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:20.374035 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:20.386661 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:20.386637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:20.644469 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:20.644363 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw" Jul 29 07:24:59.852178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.850329 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z"] Jul 29 07:24:59.853700 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.853678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.861266 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:24:59.861237 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" podUID="d49480a1-37bb-49cb-a25d-260f0933d697" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.861518 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:24:59.861498 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-143-226.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-143-226.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" Jul 29 07:24:59.917390 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.917363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z"] Jul 29 07:24:59.957196 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.957175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.957314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.957208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49480a1-37bb-49cb-a25d-260f0933d697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.957314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.957233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.957314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.957275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nn7\" (UniqueName: \"kubernetes.io/projected/d49480a1-37bb-49cb-a25d-260f0933d697-kube-api-access-v6nn7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.957314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.957303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:24:59.957509 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:24:59.957368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058393 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49480a1-37bb-49cb-a25d-260f0933d697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058393 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058393 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nn7\" (UniqueName: \"kubernetes.io/projected/d49480a1-37bb-49cb-a25d-260f0933d697-kube-api-access-v6nn7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058393 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058570 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058741 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058787 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.058827 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.058797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.060394 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.060378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d49480a1-37bb-49cb-a25d-260f0933d697-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:00.076274 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:00.076247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nn7\" (UniqueName: \"kubernetes.io/projected/d49480a1-37bb-49cb-a25d-260f0933d697-kube-api-access-v6nn7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:01.058890 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:25:01.058855 2565 secret.go:189] Couldn't get secret llm/e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 29 07:25:01.059355 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:25:01.058937 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49480a1-37bb-49cb-a25d-260f0933d697-tls-certs podName:d49480a1-37bb-49cb-a25d-260f0933d697 nodeName:}" failed. No retries permitted until 2026-07-29 07:25:01.558916882 +0000 UTC m=+583.165094970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d49480a1-37bb-49cb-a25d-260f0933d697-tls-certs") pod "e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" (UID: "d49480a1-37bb-49cb-a25d-260f0933d697") : failed to sync secret cache: timed out waiting for the condition Jul 29 07:25:01.200409 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:01.200377 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 29 07:25:01.572977 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:01.572922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49480a1-37bb-49cb-a25d-260f0933d697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:01.575363 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:01.575331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d49480a1-37bb-49cb-a25d-260f0933d697-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z\" (UID: \"d49480a1-37bb-49cb-a25d-260f0933d697\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:01.666050 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:01.666011 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:01.815349 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:01.815325 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z"] Jul 29 07:25:01.817714 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:25:01.817683 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49480a1_37bb_49cb_a25d_260f0933d697.slice/crio-d0597bcd58177e07b232a63f5c749584d4385e2eb15cbd025123dc693389b56a WatchSource:0}: Error finding container d0597bcd58177e07b232a63f5c749584d4385e2eb15cbd025123dc693389b56a: Status 404 returned error can't find the container with id d0597bcd58177e07b232a63f5c749584d4385e2eb15cbd025123dc693389b56a Jul 29 07:25:01.838546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:01.838477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" event={"ID":"d49480a1-37bb-49cb-a25d-260f0933d697","Type":"ContainerStarted","Data":"d0597bcd58177e07b232a63f5c749584d4385e2eb15cbd025123dc693389b56a"} Jul 29 07:25:02.845144 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:02.845103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" event={"ID":"d49480a1-37bb-49cb-a25d-260f0933d697","Type":"ContainerStarted","Data":"88d3abac44297a247e9ce3a188433b5b46737a4bb7521ad794f90c70123cf911"} Jul 29 07:25:02.879380 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:02.879325 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" podStartSLOduration=3.637854669 podStartE2EDuration="3.879309697s" totalImagesPullingTime="241.455028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:25:02.878324916 +0000 UTC m=+584.484503025" watchObservedRunningTime="2026-07-29 07:25:02.879309697 +0000 UTC m=+584.485487805" Jul 29 07:25:11.666364 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:11.666327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:11.666758 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:11.666658 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:11.679140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:11.679112 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:25:11.901075 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:25:11.901048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z" Jul 29 07:38:00.603925 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.603893 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv"] Jul 29 07:38:00.607241 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.607224 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.611632 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.611613 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-be281498-kserve-self-signed-certs\"" Jul 29 07:38:00.625369 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.625341 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv"] Jul 29 07:38:00.744565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.744534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tmp-dir\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.744760 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.744582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-dshm\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.744760 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.744650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-home\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.744760 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.744710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tls-certs\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.744760 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.744741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd259\" (UniqueName: \"kubernetes.io/projected/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-kube-api-access-kd259\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.744947 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.744767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-model-cache\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.845580 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tmp-dir\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.845730 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-dshm\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.845730 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-home\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.845730 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tls-certs\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.845730 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd259\" (UniqueName: \"kubernetes.io/projected/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-kube-api-access-kd259\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.845730 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-model-cache\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.846079 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.845989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tmp-dir\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.846142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.846086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-home\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.846181 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.846141 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-model-cache\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.847793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.847770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-dshm\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.848154 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.848137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tls-certs\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.856264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.856180 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd259\" (UniqueName: \"kubernetes.io/projected/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-kube-api-access-kd259\") pod \"e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:00.916481 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:00.916454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:01.059158 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:01.059130 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv"] Jul 29 07:38:01.060129 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:38:01.060104 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb450af36_c1d4_47e7_a1d0_d2b531fd32e0.slice/crio-fbd8b12ee0bfe1b130ca6757ddeb4b06d8e4862aa263d82877bd854a0533c449 WatchSource:0}: Error finding container fbd8b12ee0bfe1b130ca6757ddeb4b06d8e4862aa263d82877bd854a0533c449: Status 404 returned error can't find the container with id fbd8b12ee0bfe1b130ca6757ddeb4b06d8e4862aa263d82877bd854a0533c449 Jul 29 07:38:01.761855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:01.761814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" event={"ID":"b450af36-c1d4-47e7-a1d0-d2b531fd32e0","Type":"ContainerStarted","Data":"dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68"} Jul 29 07:38:01.762240 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:01.761862 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" event={"ID":"b450af36-c1d4-47e7-a1d0-d2b531fd32e0","Type":"ContainerStarted","Data":"fbd8b12ee0bfe1b130ca6757ddeb4b06d8e4862aa263d82877bd854a0533c449"} Jul 29 07:38:01.796008 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:01.795946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" podStartSLOduration=1.795931299 podStartE2EDuration="1.795931299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:38:01.79571668 +0000 UTC m=+1363.401894789" watchObservedRunningTime="2026-07-29 07:38:01.795931299 +0000 UTC m=+1363.402109407" Jul 29 07:38:10.917135 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:10.917102 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:10.917594 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:10.917149 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:10.929669 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:10.929643 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:11.822693 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:11.822662 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:25.847837 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:25.847800 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv"] Jul 29 07:38:40.866477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:40.866433 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" podUID="b450af36-c1d4-47e7-a1d0-d2b531fd32e0" containerName="main" containerID="cri-o://dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68" gracePeriod=45 Jul 29 07:38:41.114488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.114464 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:41.292229 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292139 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-home\") pod \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " Jul 29 07:38:41.292229 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292174 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-dshm\") pod \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " Jul 29 07:38:41.292229 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292206 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tmp-dir\") pod \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " Jul 29 07:38:41.292487 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292238 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tls-certs\") pod \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " Jul 29 07:38:41.292487 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292270 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-model-cache\") pod \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " Jul 29 07:38:41.292487 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292301 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd259\" (UniqueName: \"kubernetes.io/projected/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-kube-api-access-kd259\") pod \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\" (UID: \"b450af36-c1d4-47e7-a1d0-d2b531fd32e0\") " Jul 29 07:38:41.292487 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292455 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-home" (OuterVolumeSpecName: "home") pod "b450af36-c1d4-47e7-a1d0-d2b531fd32e0" (UID: "b450af36-c1d4-47e7-a1d0-d2b531fd32e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:38:41.292487 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292477 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b450af36-c1d4-47e7-a1d0-d2b531fd32e0" (UID: "b450af36-c1d4-47e7-a1d0-d2b531fd32e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:38:41.292745 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292571 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b450af36-c1d4-47e7-a1d0-d2b531fd32e0" (UID: "b450af36-c1d4-47e7-a1d0-d2b531fd32e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:38:41.292745 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292654 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-home\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:38:41.292745 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.292672 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tmp-dir\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:38:41.294521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.294494 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-dshm" (OuterVolumeSpecName: "dshm") pod "b450af36-c1d4-47e7-a1d0-d2b531fd32e0" (UID: "b450af36-c1d4-47e7-a1d0-d2b531fd32e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:38:41.294637 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.294558 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b450af36-c1d4-47e7-a1d0-d2b531fd32e0" (UID: "b450af36-c1d4-47e7-a1d0-d2b531fd32e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:38:41.294637 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.294579 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-kube-api-access-kd259" (OuterVolumeSpecName: "kube-api-access-kd259") pod "b450af36-c1d4-47e7-a1d0-d2b531fd32e0" (UID: "b450af36-c1d4-47e7-a1d0-d2b531fd32e0"). InnerVolumeSpecName "kube-api-access-kd259". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:38:41.393857 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.393803 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-dshm\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:38:41.393857 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.393849 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-tls-certs\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:38:41.393857 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.393859 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-model-cache\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:38:41.393857 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.393868 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd259\" (UniqueName: \"kubernetes.io/projected/b450af36-c1d4-47e7-a1d0-d2b531fd32e0-kube-api-access-kd259\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:38:41.966965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.966916 2565 generic.go:356] "Generic (PLEG): container finished" podID="b450af36-c1d4-47e7-a1d0-d2b531fd32e0" containerID="dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68" exitCode=0 Jul 29 07:38:41.967470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.966980 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" Jul 29 07:38:41.967470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.966989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" event={"ID":"b450af36-c1d4-47e7-a1d0-d2b531fd32e0","Type":"ContainerDied","Data":"dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68"} Jul 29 07:38:41.967470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.967038 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv" event={"ID":"b450af36-c1d4-47e7-a1d0-d2b531fd32e0","Type":"ContainerDied","Data":"fbd8b12ee0bfe1b130ca6757ddeb4b06d8e4862aa263d82877bd854a0533c449"} Jul 29 07:38:41.967470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.967056 2565 scope.go:117] "RemoveContainer" containerID="dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68" Jul 29 07:38:41.976466 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.976450 2565 scope.go:117] "RemoveContainer" containerID="dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68" Jul 29 07:38:41.976711 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:38:41.976691 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68\": container with ID starting with dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68 not found: ID does not exist" containerID="dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68" Jul 29 07:38:41.976757 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.976720 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68"} err="failed to get container status \"dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68\": rpc error: code = NotFound desc = could not find container \"dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68\": container with ID starting with dae4b0c5769a7e738040beaebdc4ff5f1b3f5d643ff0afc92ef90a8fd65a7c68 not found: ID does not exist" Jul 29 07:38:41.998217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:41.998191 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv"] Jul 29 07:38:42.002603 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:42.002581 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-be281498-kserve-6696556f6c-xrqwv"] Jul 29 07:38:43.018747 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:38:43.018716 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b450af36-c1d4-47e7-a1d0-d2b531fd32e0" path="/var/lib/kubelet/pods/b450af36-c1d4-47e7-a1d0-d2b531fd32e0/volumes" Jul 29 07:42:10.468779 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.468687 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5"] Jul 29 07:42:10.471193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.469070 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b450af36-c1d4-47e7-a1d0-d2b531fd32e0" containerName="main" Jul 29 07:42:10.471193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.469082 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b450af36-c1d4-47e7-a1d0-d2b531fd32e0" containerName="main" Jul 29 07:42:10.471193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.469144 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b450af36-c1d4-47e7-a1d0-d2b531fd32e0" containerName="main" Jul 29 07:42:10.472075 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.472053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.496129 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.496105 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cb4ef1f3-openshift-default-dockercfg-n78hj\"" Jul 29 07:42:10.557490 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.557459 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5"] Jul 29 07:42:10.632502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istiod-ca-cert\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.632675 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-podinfo\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.632675 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-certs\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.632801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-credential-socket\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.632801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-socket\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.632801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgfw\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-kube-api-access-2vgfw\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.633050 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-data\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.633050 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-token\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.633050 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.632866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-envoy\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733600 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-credential-socket\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733600 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-socket\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733600 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgfw\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-kube-api-access-2vgfw\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-data\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-token\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-envoy\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istiod-ca-cert\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-podinfo\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.733861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-certs\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.734217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.733944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-credential-socket\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.734217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.734030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-data\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.734217 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.734049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-socket\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.734353 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.734294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-certs\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.734491 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.734470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istiod-ca-cert\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.736169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.736149 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-envoy\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.736258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.736193 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-podinfo\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.742454 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.742435 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-token\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.742664 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.742646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgfw\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-kube-api-access-2vgfw\") pod \"e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.783971 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.783930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:10.921050 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.921020 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5"] Jul 29 07:42:10.922566 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:42:10.922508 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75679a80_d2ea_4043_86ae_bee3b1f8a4b0.slice/crio-c5632a1be333cd9fa59afa502830b3a6c34b3af83039a7f45e714077ce314461 WatchSource:0}: Error finding container c5632a1be333cd9fa59afa502830b3a6c34b3af83039a7f45e714077ce314461: Status 404 returned error can't find the container with id c5632a1be333cd9fa59afa502830b3a6c34b3af83039a7f45e714077ce314461 Jul 29 07:42:10.924772 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.924737 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:42:10.924855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.924820 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:42:10.924893 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:10.924860 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:42:11.026235 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:11.026203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" event={"ID":"75679a80-d2ea-4043-86ae-bee3b1f8a4b0","Type":"ContainerStarted","Data":"32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab"} Jul 29 07:42:11.026413 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:11.026269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" event={"ID":"75679a80-d2ea-4043-86ae-bee3b1f8a4b0","Type":"ContainerStarted","Data":"c5632a1be333cd9fa59afa502830b3a6c34b3af83039a7f45e714077ce314461"} Jul 29 07:42:11.053846 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:11.053787 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podStartSLOduration=1.053767714 podStartE2EDuration="1.053767714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:11.05276067 +0000 UTC m=+1612.658938805" watchObservedRunningTime="2026-07-29 07:42:11.053767714 +0000 UTC m=+1612.659945822" Jul 29 07:42:11.784193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:11.784161 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:11.788930 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:11.788908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:12.031708 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:12.031677 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:12.032497 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:12.032477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:13.905002 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.904967 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg"] Jul 29 07:42:13.908454 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.908434 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:13.918536 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.918515 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cb4ef1f3-dockercfg-sm746\"" Jul 29 07:42:13.918848 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.918833 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cb4ef1f3\"" Jul 29 07:42:13.933377 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.933353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg"] Jul 29 07:42:13.987294 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.987263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj"] Jul 29 07:42:13.990778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:13.990761 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.007801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.007773 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj"] Jul 29 07:42:14.065116 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.065082 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52s7\" (UniqueName: \"kubernetes.io/projected/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-kube-api-access-b52s7\") pod \"payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.065243 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.065131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgmd\" (UniqueName: \"kubernetes.io/projected/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-kube-api-access-4jgmd\") pod \"payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.065243 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.065229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.065335 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.065263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.166549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.166478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.166813 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.166787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.166943 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.166924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52s7\" (UniqueName: \"kubernetes.io/projected/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-kube-api-access-b52s7\") pod \"payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.167042 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.167004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgmd\" (UniqueName: \"kubernetes.io/projected/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-kube-api-access-4jgmd\") pod \"payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.167246 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.167227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.167506 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.167488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.181504 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.181479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52s7\" (UniqueName: \"kubernetes.io/projected/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-kube-api-access-b52s7\") pod \"payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.184963 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.184933 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgmd\" (UniqueName: \"kubernetes.io/projected/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-kube-api-access-4jgmd\") pod \"payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.218869 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.218849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:14.300273 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.300241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:14.366229 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.366200 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg"] Jul 29 07:42:14.366832 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:42:14.366806 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2d94ff_85f2_495b_a2d1_06c097b8f92f.slice/crio-f7721c6efbf1de77b26e2a9c33da6bfee5b299fe9e22baae37cb21519d0d0e8a WatchSource:0}: Error finding container f7721c6efbf1de77b26e2a9c33da6bfee5b299fe9e22baae37cb21519d0d0e8a: Status 404 returned error can't find the container with id f7721c6efbf1de77b26e2a9c33da6bfee5b299fe9e22baae37cb21519d0d0e8a Jul 29 07:42:14.472442 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:14.472417 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj"] Jul 29 07:42:14.474088 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:42:14.474062 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b81a32c_4615_4137_b3ce_4ab29d10c2b5.slice/crio-711b099733ff51c3759ed4bcc3267d34549d4ceb74a6e3546f56f38f201e432c WatchSource:0}: Error finding container 711b099733ff51c3759ed4bcc3267d34549d4ceb74a6e3546f56f38f201e432c: Status 404 returned error can't find the container with id 711b099733ff51c3759ed4bcc3267d34549d4ceb74a6e3546f56f38f201e432c Jul 29 07:42:15.050169 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.050127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" event={"ID":"2d2d94ff-85f2-495b-a2d1-06c097b8f92f","Type":"ContainerStarted","Data":"8cf2906d89d24e4765ce612ba5d09ccbe086de6309b88eddf8321210cca70000"} Jul 29 07:42:15.050681 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.050195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" event={"ID":"2d2d94ff-85f2-495b-a2d1-06c097b8f92f","Type":"ContainerStarted","Data":"f7721c6efbf1de77b26e2a9c33da6bfee5b299fe9e22baae37cb21519d0d0e8a"} Jul 29 07:42:15.050681 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.050227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:15.051797 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.051772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" event={"ID":"0b81a32c-4615-4137-b3ce-4ab29d10c2b5","Type":"ContainerStarted","Data":"fade9f81da66cb14e5ceea287c71bea42a39e62d392d021809af256a74c65ed8"} Jul 29 07:42:15.051910 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.051812 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" event={"ID":"0b81a32c-4615-4137-b3ce-4ab29d10c2b5","Type":"ContainerStarted","Data":"711b099733ff51c3759ed4bcc3267d34549d4ceb74a6e3546f56f38f201e432c"} Jul 29 07:42:15.051910 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.051833 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:15.109152 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.109105 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" podStartSLOduration=2.109089099 podStartE2EDuration="2.109089099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:15.105962861 +0000 UTC m=+1616.712140960" watchObservedRunningTime="2026-07-29 07:42:15.109089099 +0000 UTC m=+1616.715267206" Jul 29 07:42:15.160497 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:15.160436 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" podStartSLOduration=2.160416938 podStartE2EDuration="2.160416938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:15.158747797 +0000 UTC m=+1616.764925906" watchObservedRunningTime="2026-07-29 07:42:15.160416938 +0000 UTC m=+1616.766595048" Jul 29 07:42:26.061435 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:26.061398 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:26.063090 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:26.063070 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:28.074098 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:28.074059 2565 patch_prober.go:28] interesting pod/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 07:42:28.074500 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:28.074127 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 07:42:29.133929 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:29.133900 2565 generic.go:356] "Generic (PLEG): container finished" podID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerID="32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab" exitCode=0 Jul 29 07:42:29.134270 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:29.133964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" event={"ID":"75679a80-d2ea-4043-86ae-bee3b1f8a4b0","Type":"ContainerDied","Data":"32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab"} Jul 29 07:42:29.134364 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:29.134346 2565 scope.go:117] "RemoveContainer" containerID="32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab" Jul 29 07:42:29.135344 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:29.135315 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:42:29.135397 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:29.135382 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:42:29.135438 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:29.135418 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:42:30.141125 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:30.141089 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" event={"ID":"75679a80-d2ea-4043-86ae-bee3b1f8a4b0","Type":"ContainerStarted","Data":"f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83"} Jul 29 07:42:30.785130 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:30.785086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:30.785348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:30.785322 2565 patch_prober.go:28] interesting pod/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:42:30.785430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:30.785380 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 07:42:31.784984 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:31.784929 2565 patch_prober.go:28] interesting pod/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:42:31.785368 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:31.785006 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 07:42:32.784287 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:32.784251 2565 patch_prober.go:28] interesting pod/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:42:32.784478 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:32.784302 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 07:42:33.785312 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:33.785274 2565 patch_prober.go:28] interesting pod/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:42:33.785690 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:33.785342 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 07:42:34.787379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:34.787347 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:34.787759 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:34.787688 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:34.788394 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:34.788374 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:44.004911 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.004874 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj"] Jul 29 07:42:44.005771 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.005177 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" podUID="0b81a32c-4615-4137-b3ce-4ab29d10c2b5" containerName="payload-processing" containerID="cri-o://fade9f81da66cb14e5ceea287c71bea42a39e62d392d021809af256a74c65ed8" gracePeriod=30 Jul 29 07:42:44.008207 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.008180 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg"] Jul 29 07:42:44.008436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.008396 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" podUID="2d2d94ff-85f2-495b-a2d1-06c097b8f92f" containerName="payload-pre-processing" containerID="cri-o://8cf2906d89d24e4765ce612ba5d09ccbe086de6309b88eddf8321210cca70000" gracePeriod=30 Jul 29 07:42:44.231776 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.231746 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b81a32c-4615-4137-b3ce-4ab29d10c2b5" containerID="fade9f81da66cb14e5ceea287c71bea42a39e62d392d021809af256a74c65ed8" exitCode=0 Jul 29 07:42:44.231926 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.231825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" event={"ID":"0b81a32c-4615-4137-b3ce-4ab29d10c2b5","Type":"ContainerDied","Data":"fade9f81da66cb14e5ceea287c71bea42a39e62d392d021809af256a74c65ed8"} Jul 29 07:42:44.233581 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.233555 2565 generic.go:356] "Generic (PLEG): container finished" podID="2d2d94ff-85f2-495b-a2d1-06c097b8f92f" containerID="8cf2906d89d24e4765ce612ba5d09ccbe086de6309b88eddf8321210cca70000" exitCode=0 Jul 29 07:42:44.233687 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.233583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" event={"ID":"2d2d94ff-85f2-495b-a2d1-06c097b8f92f","Type":"ContainerDied","Data":"8cf2906d89d24e4765ce612ba5d09ccbe086de6309b88eddf8321210cca70000"} Jul 29 07:42:44.287657 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.287634 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:44.291364 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.291343 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:44.349915 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.349890 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-plugins-config-volume\") pod \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " Jul 29 07:42:44.350093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.349925 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b52s7\" (UniqueName: \"kubernetes.io/projected/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-kube-api-access-b52s7\") pod \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\" (UID: \"2d2d94ff-85f2-495b-a2d1-06c097b8f92f\") " Jul 29 07:42:44.350093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.350003 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-plugins-config-volume\") pod \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " Jul 29 07:42:44.350093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.350034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jgmd\" (UniqueName: \"kubernetes.io/projected/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-kube-api-access-4jgmd\") pod \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\" (UID: \"0b81a32c-4615-4137-b3ce-4ab29d10c2b5\") " Jul 29 07:42:44.350340 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.350311 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d2d94ff-85f2-495b-a2d1-06c097b8f92f" (UID: "2d2d94ff-85f2-495b-a2d1-06c097b8f92f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:42:44.350402 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.350381 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b81a32c-4615-4137-b3ce-4ab29d10c2b5" (UID: "0b81a32c-4615-4137-b3ce-4ab29d10c2b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:42:44.352187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.352166 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-kube-api-access-b52s7" (OuterVolumeSpecName: "kube-api-access-b52s7") pod "2d2d94ff-85f2-495b-a2d1-06c097b8f92f" (UID: "2d2d94ff-85f2-495b-a2d1-06c097b8f92f"). InnerVolumeSpecName "kube-api-access-b52s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:42:44.352300 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.352246 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-kube-api-access-4jgmd" (OuterVolumeSpecName: "kube-api-access-4jgmd") pod "0b81a32c-4615-4137-b3ce-4ab29d10c2b5" (UID: "0b81a32c-4615-4137-b3ce-4ab29d10c2b5"). InnerVolumeSpecName "kube-api-access-4jgmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:42:44.451238 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.451211 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.451238 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.451235 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b52s7\" (UniqueName: \"kubernetes.io/projected/2d2d94ff-85f2-495b-a2d1-06c097b8f92f-kube-api-access-b52s7\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.451414 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.451245 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.451414 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.451254 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jgmd\" (UniqueName: \"kubernetes.io/projected/0b81a32c-4615-4137-b3ce-4ab29d10c2b5-kube-api-access-4jgmd\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:44.628267 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.628235 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5"] Jul 29 07:42:44.628456 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:44.628431 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" containerID="cri-o://f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83" gracePeriod=30 Jul 29 07:42:45.240224 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.240183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" Jul 29 07:42:45.240224 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.240204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg" event={"ID":"2d2d94ff-85f2-495b-a2d1-06c097b8f92f","Type":"ContainerDied","Data":"f7721c6efbf1de77b26e2a9c33da6bfee5b299fe9e22baae37cb21519d0d0e8a"} Jul 29 07:42:45.240724 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.240271 2565 scope.go:117] "RemoveContainer" containerID="8cf2906d89d24e4765ce612ba5d09ccbe086de6309b88eddf8321210cca70000" Jul 29 07:42:45.241594 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.241568 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" event={"ID":"0b81a32c-4615-4137-b3ce-4ab29d10c2b5","Type":"ContainerDied","Data":"711b099733ff51c3759ed4bcc3267d34549d4ceb74a6e3546f56f38f201e432c"} Jul 29 07:42:45.241672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.241635 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj" Jul 29 07:42:45.249965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.249936 2565 scope.go:117] "RemoveContainer" containerID="fade9f81da66cb14e5ceea287c71bea42a39e62d392d021809af256a74c65ed8" Jul 29 07:42:45.262984 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.262946 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg"] Jul 29 07:42:45.271683 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.271663 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cb4ef1f3-5fb65ddfd4-k4mzg"] Jul 29 07:42:45.285637 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.285581 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj"] Jul 29 07:42:45.288259 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:45.288238 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cb4ef1f3-79cb985888-67dcj"] Jul 29 07:42:47.019632 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:47.019595 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b81a32c-4615-4137-b3ce-4ab29d10c2b5" path="/var/lib/kubelet/pods/0b81a32c-4615-4137-b3ce-4ab29d10c2b5/volumes" Jul 29 07:42:47.020024 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:47.019968 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2d94ff-85f2-495b-a2d1-06c097b8f92f" path="/var/lib/kubelet/pods/2d2d94ff-85f2-495b-a2d1-06c097b8f92f/volumes" Jul 29 07:42:49.789157 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.789125 2565 patch_prober.go:28] interesting pod/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 29 07:42:49.789484 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.789177 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 29 07:42:49.883806 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.883780 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:49.993191 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-credential-socket\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993191 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993152 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-socket\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993206 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-envoy\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993233 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istiod-ca-cert\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993260 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-data\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993292 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-token\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993430 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:42:49.993610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993493 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-podinfo\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993560 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:42:49.993778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993616 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vgfw\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-kube-api-access-2vgfw\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993643 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:42:49.993778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993647 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-certs\") pod \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\" (UID: \"75679a80-d2ea-4043-86ae-bee3b1f8a4b0\") " Jul 29 07:42:49.993921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.993846 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:42:49.994195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.994138 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-certs\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:49.994195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.994164 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-credential-socket\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:49.994195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.994170 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:42:49.994195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.994178 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-workload-socket\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:49.994508 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.994220 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istiod-ca-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:49.995652 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.995623 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 07:42:49.995753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.995711 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 07:42:49.995753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.995720 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-kube-api-access-2vgfw" (OuterVolumeSpecName: "kube-api-access-2vgfw") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "kube-api-access-2vgfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:42:49.995753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:49.995731 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "75679a80-d2ea-4043-86ae-bee3b1f8a4b0" (UID: "75679a80-d2ea-4043-86ae-bee3b1f8a4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:42:50.094794 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.094766 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-envoy\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:50.094794 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.094789 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-data\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:50.094794 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.094798 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-token\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:50.095193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.094808 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-istio-podinfo\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:50.095193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.094816 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgfw\" (UniqueName: \"kubernetes.io/projected/75679a80-d2ea-4043-86ae-bee3b1f8a4b0-kube-api-access-2vgfw\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:42:50.269089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.269016 2565 generic.go:356] "Generic (PLEG): container finished" podID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerID="f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83" exitCode=0 Jul 29 07:42:50.269089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.269074 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" Jul 29 07:42:50.269258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.269101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" event={"ID":"75679a80-d2ea-4043-86ae-bee3b1f8a4b0","Type":"ContainerDied","Data":"f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83"} Jul 29 07:42:50.269258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.269155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5" event={"ID":"75679a80-d2ea-4043-86ae-bee3b1f8a4b0","Type":"ContainerDied","Data":"c5632a1be333cd9fa59afa502830b3a6c34b3af83039a7f45e714077ce314461"} Jul 29 07:42:50.269258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.269178 2565 scope.go:117] "RemoveContainer" containerID="f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83" Jul 29 07:42:50.277861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.277846 2565 scope.go:117] "RemoveContainer" containerID="32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab" Jul 29 07:42:50.285383 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.285331 2565 scope.go:117] "RemoveContainer" containerID="f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83" Jul 29 07:42:50.285597 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:42:50.285581 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83\": container with ID starting with f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83 not found: ID does not exist" containerID="f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83" Jul 29 07:42:50.285645 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.285604 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83"} err="failed to get container status \"f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83\": rpc error: code = NotFound desc = could not find container \"f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83\": container with ID starting with f2f9fb914fdbcd36b4eaeffc394a16cc7b4d0ef4d98c7bdf7bbdf203d822ea83 not found: ID does not exist" Jul 29 07:42:50.285645 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.285619 2565 scope.go:117] "RemoveContainer" containerID="32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab" Jul 29 07:42:50.285844 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:42:50.285827 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab\": container with ID starting with 32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab not found: ID does not exist" containerID="32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab" Jul 29 07:42:50.285882 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.285850 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab"} err="failed to get container status \"32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab\": rpc error: code = NotFound desc = could not find container \"32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab\": container with ID starting with 32d3199a806ee954daaa1635093f178c71fb5864abe897d5049389f3e4c463ab not found: ID does not exist" Jul 29 07:42:50.301145 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.301124 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5"] Jul 29 07:42:50.310332 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:50.310313 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cb4ef1f3-openshift-default-86d7564cfb-pzzm5"] Jul 29 07:42:51.019826 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:51.019791 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" path="/var/lib/kubelet/pods/75679a80-d2ea-4043-86ae-bee3b1f8a4b0/volumes" Jul 29 07:42:52.438359 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438325 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76"] Jul 29 07:42:52.438725 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438694 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2d94ff-85f2-495b-a2d1-06c097b8f92f" containerName="payload-pre-processing" Jul 29 07:42:52.438725 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438708 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2d94ff-85f2-495b-a2d1-06c097b8f92f" containerName="payload-pre-processing" Jul 29 07:42:52.438725 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438717 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" Jul 29 07:42:52.438725 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438722 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" Jul 29 07:42:52.438855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438737 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b81a32c-4615-4137-b3ce-4ab29d10c2b5" containerName="payload-processing" Jul 29 07:42:52.438855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438743 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b81a32c-4615-4137-b3ce-4ab29d10c2b5" containerName="payload-processing" Jul 29 07:42:52.438855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438807 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" Jul 29 07:42:52.438855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438814 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" Jul 29 07:42:52.438855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438822 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b81a32c-4615-4137-b3ce-4ab29d10c2b5" containerName="payload-processing" Jul 29 07:42:52.438855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.438836 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2d94ff-85f2-495b-a2d1-06c097b8f92f" containerName="payload-pre-processing" Jul 29 07:42:52.441991 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.441973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.459280 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.459259 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8672bcd3-dockercfg-b5x56\"" Jul 29 07:42:52.459389 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.459352 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8672bcd3\"" Jul 29 07:42:52.463167 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.463146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76"] Jul 29 07:42:52.516904 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.516877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7452c80d-4def-4bf8-b8a6-7a757f409bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.517029 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.516935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcdff\" (UniqueName: \"kubernetes.io/projected/7452c80d-4def-4bf8-b8a6-7a757f409bce-kube-api-access-rcdff\") pod \"payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.524588 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.524566 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7"] Jul 29 07:42:52.524907 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.524896 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" Jul 29 07:42:52.524946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.524908 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75679a80-d2ea-4043-86ae-bee3b1f8a4b0" containerName="istio-proxy" Jul 29 07:42:52.526922 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.526908 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.559013 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.558976 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7"] Jul 29 07:42:52.617624 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.617588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d064bf09-c961-440a-9bb0-b46e5f463a79-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.617808 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.617656 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7452c80d-4def-4bf8-b8a6-7a757f409bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.617808 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.617753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4lr\" (UniqueName: \"kubernetes.io/projected/d064bf09-c961-440a-9bb0-b46e5f463a79-kube-api-access-7x4lr\") pod \"payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.617808 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.617802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcdff\" (UniqueName: \"kubernetes.io/projected/7452c80d-4def-4bf8-b8a6-7a757f409bce-kube-api-access-rcdff\") pod \"payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.618329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.618310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7452c80d-4def-4bf8-b8a6-7a757f409bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.649379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.649359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcdff\" (UniqueName: \"kubernetes.io/projected/7452c80d-4def-4bf8-b8a6-7a757f409bce-kube-api-access-rcdff\") pod \"payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.718940 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.718879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d064bf09-c961-440a-9bb0-b46e5f463a79-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.719062 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.718944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4lr\" (UniqueName: \"kubernetes.io/projected/d064bf09-c961-440a-9bb0-b46e5f463a79-kube-api-access-7x4lr\") pod \"payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.719580 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.719557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d064bf09-c961-440a-9bb0-b46e5f463a79-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.730120 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.730095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4lr\" (UniqueName: \"kubernetes.io/projected/d064bf09-c961-440a-9bb0-b46e5f463a79-kube-api-access-7x4lr\") pod \"payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.752088 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.752067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:52.837502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.837275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:52.886740 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.886707 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76"] Jul 29 07:42:52.887897 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:42:52.887870 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7452c80d_4def_4bf8_b8a6_7a757f409bce.slice/crio-7626ad9323bef7cc6449b6533554e0bb4388d1551f360fbb6c9630749cf7214d WatchSource:0}: Error finding container 7626ad9323bef7cc6449b6533554e0bb4388d1551f360fbb6c9630749cf7214d: Status 404 returned error can't find the container with id 7626ad9323bef7cc6449b6533554e0bb4388d1551f360fbb6c9630749cf7214d Jul 29 07:42:52.991466 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:42:52.991439 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd064bf09_c961_440a_9bb0_b46e5f463a79.slice/crio-81e9f93c85826eb95c6cc0dfd751a05da5a54b9e897bb28e8eed22f4ec27337b WatchSource:0}: Error finding container 81e9f93c85826eb95c6cc0dfd751a05da5a54b9e897bb28e8eed22f4ec27337b: Status 404 returned error can't find the container with id 81e9f93c85826eb95c6cc0dfd751a05da5a54b9e897bb28e8eed22f4ec27337b Jul 29 07:42:52.993320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:52.993298 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7"] Jul 29 07:42:53.288076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.287988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" event={"ID":"d064bf09-c961-440a-9bb0-b46e5f463a79","Type":"ContainerStarted","Data":"588b7f6cdb3c143ae8510bddb281331284fb2029492ed302ad34ec04135910c0"} Jul 29 07:42:53.288076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.288060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" event={"ID":"d064bf09-c961-440a-9bb0-b46e5f463a79","Type":"ContainerStarted","Data":"81e9f93c85826eb95c6cc0dfd751a05da5a54b9e897bb28e8eed22f4ec27337b"} Jul 29 07:42:53.288304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.288108 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:42:53.289559 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.289537 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" event={"ID":"7452c80d-4def-4bf8-b8a6-7a757f409bce","Type":"ContainerStarted","Data":"1f308b730e5f81039837259ba085e4c9d81ef2360338d994880148f429a7e0d8"} Jul 29 07:42:53.289672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.289574 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" event={"ID":"7452c80d-4def-4bf8-b8a6-7a757f409bce","Type":"ContainerStarted","Data":"7626ad9323bef7cc6449b6533554e0bb4388d1551f360fbb6c9630749cf7214d"} Jul 29 07:42:53.289672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.289662 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:42:53.320761 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:42:53.320717 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" podStartSLOduration=1.320702448 podStartE2EDuration="1.320702448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:53.319015253 +0000 UTC m=+1654.925193361" watchObservedRunningTime="2026-07-29 07:42:53.320702448 +0000 UTC m=+1654.926880555" Jul 29 07:43:04.298588 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:04.298547 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:43:04.299019 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:04.298768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:43:04.332932 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:04.332888 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" podStartSLOduration=12.332874949 podStartE2EDuration="12.332874949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:42:53.357852746 +0000 UTC m=+1654.964030854" watchObservedRunningTime="2026-07-29 07:43:04.332874949 +0000 UTC m=+1665.939053057" Jul 29 07:43:22.273836 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.273801 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7"] Jul 29 07:43:22.274363 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.274054 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" podUID="d064bf09-c961-440a-9bb0-b46e5f463a79" containerName="payload-processing" containerID="cri-o://588b7f6cdb3c143ae8510bddb281331284fb2029492ed302ad34ec04135910c0" gracePeriod=30 Jul 29 07:43:22.306737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.306712 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76"] Jul 29 07:43:22.306936 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.306901 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" podUID="7452c80d-4def-4bf8-b8a6-7a757f409bce" containerName="payload-pre-processing" containerID="cri-o://1f308b730e5f81039837259ba085e4c9d81ef2360338d994880148f429a7e0d8" gracePeriod=30 Jul 29 07:43:22.447594 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.447563 2565 generic.go:356] "Generic (PLEG): container finished" podID="7452c80d-4def-4bf8-b8a6-7a757f409bce" containerID="1f308b730e5f81039837259ba085e4c9d81ef2360338d994880148f429a7e0d8" exitCode=0 Jul 29 07:43:22.447793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.447646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" event={"ID":"7452c80d-4def-4bf8-b8a6-7a757f409bce","Type":"ContainerDied","Data":"1f308b730e5f81039837259ba085e4c9d81ef2360338d994880148f429a7e0d8"} Jul 29 07:43:22.449519 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.449482 2565 generic.go:356] "Generic (PLEG): container finished" podID="d064bf09-c961-440a-9bb0-b46e5f463a79" containerID="588b7f6cdb3c143ae8510bddb281331284fb2029492ed302ad34ec04135910c0" exitCode=0 Jul 29 07:43:22.449651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.449578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" event={"ID":"d064bf09-c961-440a-9bb0-b46e5f463a79","Type":"ContainerDied","Data":"588b7f6cdb3c143ae8510bddb281331284fb2029492ed302ad34ec04135910c0"} Jul 29 07:43:22.570504 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.570469 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:43:22.609589 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.609559 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:43:22.662232 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.662208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcdff\" (UniqueName: \"kubernetes.io/projected/7452c80d-4def-4bf8-b8a6-7a757f409bce-kube-api-access-rcdff\") pod \"7452c80d-4def-4bf8-b8a6-7a757f409bce\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " Jul 29 07:43:22.662396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.662254 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d064bf09-c961-440a-9bb0-b46e5f463a79-plugins-config-volume\") pod \"d064bf09-c961-440a-9bb0-b46e5f463a79\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " Jul 29 07:43:22.662396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.662314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x4lr\" (UniqueName: \"kubernetes.io/projected/d064bf09-c961-440a-9bb0-b46e5f463a79-kube-api-access-7x4lr\") pod \"d064bf09-c961-440a-9bb0-b46e5f463a79\" (UID: \"d064bf09-c961-440a-9bb0-b46e5f463a79\") " Jul 29 07:43:22.662396 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.662339 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7452c80d-4def-4bf8-b8a6-7a757f409bce-plugins-config-volume\") pod \"7452c80d-4def-4bf8-b8a6-7a757f409bce\" (UID: \"7452c80d-4def-4bf8-b8a6-7a757f409bce\") " Jul 29 07:43:22.662719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.662685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d064bf09-c961-440a-9bb0-b46e5f463a79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d064bf09-c961-440a-9bb0-b46e5f463a79" (UID: "d064bf09-c961-440a-9bb0-b46e5f463a79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:22.662719 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.662713 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7452c80d-4def-4bf8-b8a6-7a757f409bce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7452c80d-4def-4bf8-b8a6-7a757f409bce" (UID: "7452c80d-4def-4bf8-b8a6-7a757f409bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:22.664292 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.664269 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7452c80d-4def-4bf8-b8a6-7a757f409bce-kube-api-access-rcdff" (OuterVolumeSpecName: "kube-api-access-rcdff") pod "7452c80d-4def-4bf8-b8a6-7a757f409bce" (UID: "7452c80d-4def-4bf8-b8a6-7a757f409bce"). InnerVolumeSpecName "kube-api-access-rcdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:22.664408 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.664306 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d064bf09-c961-440a-9bb0-b46e5f463a79-kube-api-access-7x4lr" (OuterVolumeSpecName: "kube-api-access-7x4lr") pod "d064bf09-c961-440a-9bb0-b46e5f463a79" (UID: "d064bf09-c961-440a-9bb0-b46e5f463a79"). InnerVolumeSpecName "kube-api-access-7x4lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:22.763685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.763657 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d064bf09-c961-440a-9bb0-b46e5f463a79-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:22.763685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.763680 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4lr\" (UniqueName: \"kubernetes.io/projected/d064bf09-c961-440a-9bb0-b46e5f463a79-kube-api-access-7x4lr\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:22.763685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.763690 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7452c80d-4def-4bf8-b8a6-7a757f409bce-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:22.763882 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:22.763699 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcdff\" (UniqueName: \"kubernetes.io/projected/7452c80d-4def-4bf8-b8a6-7a757f409bce-kube-api-access-rcdff\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:23.456491 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.456463 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" Jul 29 07:43:23.456932 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.456493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76" event={"ID":"7452c80d-4def-4bf8-b8a6-7a757f409bce","Type":"ContainerDied","Data":"7626ad9323bef7cc6449b6533554e0bb4388d1551f360fbb6c9630749cf7214d"} Jul 29 07:43:23.456932 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.456556 2565 scope.go:117] "RemoveContainer" containerID="1f308b730e5f81039837259ba085e4c9d81ef2360338d994880148f429a7e0d8" Jul 29 07:43:23.457921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.457857 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" Jul 29 07:43:23.457921 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.457904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7" event={"ID":"d064bf09-c961-440a-9bb0-b46e5f463a79","Type":"ContainerDied","Data":"81e9f93c85826eb95c6cc0dfd751a05da5a54b9e897bb28e8eed22f4ec27337b"} Jul 29 07:43:23.465574 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.465557 2565 scope.go:117] "RemoveContainer" containerID="588b7f6cdb3c143ae8510bddb281331284fb2029492ed302ad34ec04135910c0" Jul 29 07:43:23.511784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.511762 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7"] Jul 29 07:43:23.521333 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.521313 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8672bcd3-7b7bf676b5-ngfz7"] Jul 29 07:43:23.552081 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.552028 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76"] Jul 29 07:43:23.562356 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:23.562325 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8672bcd3-59798879dd-6tr76"] Jul 29 07:43:25.018991 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:25.018943 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7452c80d-4def-4bf8-b8a6-7a757f409bce" path="/var/lib/kubelet/pods/7452c80d-4def-4bf8-b8a6-7a757f409bce/volumes" Jul 29 07:43:25.019371 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:25.019327 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d064bf09-c961-440a-9bb0-b46e5f463a79" path="/var/lib/kubelet/pods/d064bf09-c961-440a-9bb0-b46e5f463a79/volumes" Jul 29 07:43:26.749546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.749514 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9"] Jul 29 07:43:26.749903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.749897 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7452c80d-4def-4bf8-b8a6-7a757f409bce" containerName="payload-pre-processing" Jul 29 07:43:26.749946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.749908 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7452c80d-4def-4bf8-b8a6-7a757f409bce" containerName="payload-pre-processing" Jul 29 07:43:26.749946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.749917 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d064bf09-c961-440a-9bb0-b46e5f463a79" containerName="payload-processing" Jul 29 07:43:26.749946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.749922 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d064bf09-c961-440a-9bb0-b46e5f463a79" containerName="payload-processing" Jul 29 07:43:26.750084 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.750006 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d064bf09-c961-440a-9bb0-b46e5f463a79" containerName="payload-processing" Jul 29 07:43:26.750084 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.750016 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7452c80d-4def-4bf8-b8a6-7a757f409bce" containerName="payload-pre-processing" Jul 29 07:43:26.753201 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.753185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.756224 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.756194 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:43:26.756365 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.756273 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:43:26.756365 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.756307 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:43:26.756365 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.756327 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d04dd228-serving-cert\"" Jul 29 07:43:26.773270 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.773245 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9"] Jul 29 07:43:26.802651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.802630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.802767 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.802693 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9ss\" (UniqueName: \"kubernetes.io/projected/f2fc98c7-d83f-488f-9cef-e72263b14ae1-kube-api-access-gj9ss\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.802814 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.802780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc98c7-d83f-488f-9cef-e72263b14ae1-openshift-service-ca\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.904046 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.904013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9ss\" (UniqueName: \"kubernetes.io/projected/f2fc98c7-d83f-488f-9cef-e72263b14ae1-kube-api-access-gj9ss\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.904247 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.904072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc98c7-d83f-488f-9cef-e72263b14ae1-openshift-service-ca\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.904247 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.904108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.904247 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:43:26.904225 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-serving-cert: secret "maas-api-e2e-ait-d04dd228-serving-cert" not found Jul 29 07:43:26.904439 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:43:26.904288 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls podName:f2fc98c7-d83f-488f-9cef-e72263b14ae1 nodeName:}" failed. No retries permitted until 2026-07-29 07:43:27.404272044 +0000 UTC m=+1689.010450131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls") pod "maas-api-e2e-ait-d04dd228-67f57df474-595k9" (UID: "f2fc98c7-d83f-488f-9cef-e72263b14ae1") : secret "maas-api-e2e-ait-d04dd228-serving-cert" not found Jul 29 07:43:26.904701 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.904682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc98c7-d83f-488f-9cef-e72263b14ae1-openshift-service-ca\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:26.917076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:26.917051 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9ss\" (UniqueName: \"kubernetes.io/projected/f2fc98c7-d83f-488f-9cef-e72263b14ae1-kube-api-access-gj9ss\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:27.408614 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:27.408585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:27.410924 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:27.410901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls\") pod \"maas-api-e2e-ait-d04dd228-67f57df474-595k9\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:27.662627 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:27.662555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:27.829745 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:43:27.829718 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fc98c7_d83f_488f_9cef_e72263b14ae1.slice/crio-432e5f0c0f4b1e88fb4c5477fa78e393910e34ff4671a058c4ef7aae0e920d20 WatchSource:0}: Error finding container 432e5f0c0f4b1e88fb4c5477fa78e393910e34ff4671a058c4ef7aae0e920d20: Status 404 returned error can't find the container with id 432e5f0c0f4b1e88fb4c5477fa78e393910e34ff4671a058c4ef7aae0e920d20 Jul 29 07:43:27.830940 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:27.830914 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9"] Jul 29 07:43:27.831099 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:27.831083 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:43:28.488433 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.488396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" event={"ID":"f2fc98c7-d83f-488f-9cef-e72263b14ae1","Type":"ContainerStarted","Data":"432e5f0c0f4b1e88fb4c5477fa78e393910e34ff4671a058c4ef7aae0e920d20"} Jul 29 07:43:28.617470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.617434 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8"] Jul 29 07:43:28.620746 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.620724 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.634692 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.634669 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d04dd228\"" Jul 29 07:43:28.634817 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.634674 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d04dd228-dockercfg-gctj7\"" Jul 29 07:43:28.705260 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.705230 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9"] Jul 29 07:43:28.707843 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.707822 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:28.722080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.722059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca45cb-e628-4b6d-a406-b375f2cb63eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.722195 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.722128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8nhm\" (UniqueName: \"kubernetes.io/projected/52ca45cb-e628-4b6d-a406-b375f2cb63eb-kube-api-access-p8nhm\") pod \"payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.732763 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.732736 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8"] Jul 29 07:43:28.736284 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.736260 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9"] Jul 29 07:43:28.822889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.822855 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca45cb-e628-4b6d-a406-b375f2cb63eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.823171 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.822969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spspb\" (UniqueName: \"kubernetes.io/projected/08f0c8a1-7172-440d-b534-75dfbb74be47-kube-api-access-spspb\") pod \"payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:28.823171 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.823016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8nhm\" (UniqueName: \"kubernetes.io/projected/52ca45cb-e628-4b6d-a406-b375f2cb63eb-kube-api-access-p8nhm\") pod \"payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.823355 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.823247 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f0c8a1-7172-440d-b534-75dfbb74be47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:28.823789 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.823766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca45cb-e628-4b6d-a406-b375f2cb63eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.838765 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.838738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8nhm\" (UniqueName: \"kubernetes.io/projected/52ca45cb-e628-4b6d-a406-b375f2cb63eb-kube-api-access-p8nhm\") pod \"payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.923877 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.923845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f0c8a1-7172-440d-b534-75dfbb74be47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:28.924048 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.923992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spspb\" (UniqueName: \"kubernetes.io/projected/08f0c8a1-7172-440d-b534-75dfbb74be47-kube-api-access-spspb\") pod \"payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:28.924622 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.924600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f0c8a1-7172-440d-b534-75dfbb74be47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:28.930712 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.930691 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:28.945446 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:28.945421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spspb\" (UniqueName: \"kubernetes.io/projected/08f0c8a1-7172-440d-b534-75dfbb74be47-kube-api-access-spspb\") pod \"payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:29.020291 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.020266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:29.103865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.103836 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8"] Jul 29 07:43:29.105359 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:43:29.105330 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ca45cb_e628_4b6d_a406_b375f2cb63eb.slice/crio-536cdff8a815cb721fe522484ca409d1f87e8bd31886a8a77dbbdbce1608acd7 WatchSource:0}: Error finding container 536cdff8a815cb721fe522484ca409d1f87e8bd31886a8a77dbbdbce1608acd7: Status 404 returned error can't find the container with id 536cdff8a815cb721fe522484ca409d1f87e8bd31886a8a77dbbdbce1608acd7 Jul 29 07:43:29.214628 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.214598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9"] Jul 29 07:43:29.496894 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.496799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" event={"ID":"52ca45cb-e628-4b6d-a406-b375f2cb63eb","Type":"ContainerStarted","Data":"59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21"} Jul 29 07:43:29.497120 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.497093 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:29.497244 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.497136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" event={"ID":"52ca45cb-e628-4b6d-a406-b375f2cb63eb","Type":"ContainerStarted","Data":"536cdff8a815cb721fe522484ca409d1f87e8bd31886a8a77dbbdbce1608acd7"} Jul 29 07:43:29.521098 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:29.521047 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" podStartSLOduration=1.5210269090000001 podStartE2EDuration="1.521026909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:43:29.517159087 +0000 UTC m=+1691.123337201" watchObservedRunningTime="2026-07-29 07:43:29.521026909 +0000 UTC m=+1691.127205018" Jul 29 07:43:29.552187 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:43:29.552157 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f0c8a1_7172_440d_b534_75dfbb74be47.slice/crio-810d018b78dfe5b3aa1278448317a1603f2dc9c3301ca0323404e590db485c0b WatchSource:0}: Error finding container 810d018b78dfe5b3aa1278448317a1603f2dc9c3301ca0323404e590db485c0b: Status 404 returned error can't find the container with id 810d018b78dfe5b3aa1278448317a1603f2dc9c3301ca0323404e590db485c0b Jul 29 07:43:30.503264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:30.503226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" event={"ID":"f2fc98c7-d83f-488f-9cef-e72263b14ae1","Type":"ContainerStarted","Data":"91995636a293cbc397b5aae79d1b15d4f706ca8757dee320231dd0da88d28548"} Jul 29 07:43:30.503697 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:30.503321 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:30.504808 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:30.504783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" event={"ID":"08f0c8a1-7172-440d-b534-75dfbb74be47","Type":"ContainerStarted","Data":"8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec"} Jul 29 07:43:30.504929 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:30.504828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" event={"ID":"08f0c8a1-7172-440d-b534-75dfbb74be47","Type":"ContainerStarted","Data":"810d018b78dfe5b3aa1278448317a1603f2dc9c3301ca0323404e590db485c0b"} Jul 29 07:43:30.529468 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:30.529424 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" podStartSLOduration=2.775508395 podStartE2EDuration="4.529406862s" totalImagesPullingTime="1.753898467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:43:30.52919149 +0000 UTC m=+1692.135369588" watchObservedRunningTime="2026-07-29 07:43:30.529406862 +0000 UTC m=+1692.135584970" Jul 29 07:43:30.580134 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:30.580081 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" podStartSLOduration=2.580064028 podStartE2EDuration="2.580064028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:43:30.578550121 +0000 UTC m=+1692.184728228" watchObservedRunningTime="2026-07-29 07:43:30.580064028 +0000 UTC m=+1692.186242137" Jul 29 07:43:31.510196 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:31.510161 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:36.515740 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:36.515707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:40.507678 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:40.507598 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:42.517883 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:42.517853 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:57.486754 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.486721 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9"] Jul 29 07:43:57.487347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.486941 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" podUID="f2fc98c7-d83f-488f-9cef-e72263b14ae1" containerName="maas-api" containerID="cri-o://91995636a293cbc397b5aae79d1b15d4f706ca8757dee320231dd0da88d28548" gracePeriod=30 Jul 29 07:43:57.553729 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.553692 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8"] Jul 29 07:43:57.553974 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.553931 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" podUID="52ca45cb-e628-4b6d-a406-b375f2cb63eb" containerName="payload-processing" containerID="cri-o://59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21" gracePeriod=30 Jul 29 07:43:57.562698 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.562661 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9"] Jul 29 07:43:57.563125 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.563072 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" podUID="08f0c8a1-7172-440d-b534-75dfbb74be47" containerName="payload-pre-processing" containerID="cri-o://8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec" gracePeriod=30 Jul 29 07:43:57.650600 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.650572 2565 generic.go:356] "Generic (PLEG): container finished" podID="f2fc98c7-d83f-488f-9cef-e72263b14ae1" containerID="91995636a293cbc397b5aae79d1b15d4f706ca8757dee320231dd0da88d28548" exitCode=0 Jul 29 07:43:57.650745 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.650640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" event={"ID":"f2fc98c7-d83f-488f-9cef-e72263b14ae1","Type":"ContainerDied","Data":"91995636a293cbc397b5aae79d1b15d4f706ca8757dee320231dd0da88d28548"} Jul 29 07:43:57.819650 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.819627 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:57.840861 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.840834 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:57.876199 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.876173 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:57.889671 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.889649 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls\") pod \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " Jul 29 07:43:57.889797 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.889693 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc98c7-d83f-488f-9cef-e72263b14ae1-openshift-service-ca\") pod \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " Jul 29 07:43:57.889797 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.889780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca45cb-e628-4b6d-a406-b375f2cb63eb-plugins-config-volume\") pod \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " Jul 29 07:43:57.889913 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.889800 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj9ss\" (UniqueName: \"kubernetes.io/projected/f2fc98c7-d83f-488f-9cef-e72263b14ae1-kube-api-access-gj9ss\") pod \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\" (UID: \"f2fc98c7-d83f-488f-9cef-e72263b14ae1\") " Jul 29 07:43:57.889913 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.889839 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8nhm\" (UniqueName: \"kubernetes.io/projected/52ca45cb-e628-4b6d-a406-b375f2cb63eb-kube-api-access-p8nhm\") pod \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\" (UID: \"52ca45cb-e628-4b6d-a406-b375f2cb63eb\") " Jul 29 07:43:57.890160 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.890127 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fc98c7-d83f-488f-9cef-e72263b14ae1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2fc98c7-d83f-488f-9cef-e72263b14ae1" (UID: "f2fc98c7-d83f-488f-9cef-e72263b14ae1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:57.890266 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.890222 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ca45cb-e628-4b6d-a406-b375f2cb63eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52ca45cb-e628-4b6d-a406-b375f2cb63eb" (UID: "52ca45cb-e628-4b6d-a406-b375f2cb63eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:57.890266 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.890238 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2fc98c7-d83f-488f-9cef-e72263b14ae1-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:57.891826 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.891804 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ca45cb-e628-4b6d-a406-b375f2cb63eb-kube-api-access-p8nhm" (OuterVolumeSpecName: "kube-api-access-p8nhm") pod "52ca45cb-e628-4b6d-a406-b375f2cb63eb" (UID: "52ca45cb-e628-4b6d-a406-b375f2cb63eb"). InnerVolumeSpecName "kube-api-access-p8nhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:57.891826 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.891818 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2fc98c7-d83f-488f-9cef-e72263b14ae1" (UID: "f2fc98c7-d83f-488f-9cef-e72263b14ae1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:43:57.891931 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.891878 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fc98c7-d83f-488f-9cef-e72263b14ae1-kube-api-access-gj9ss" (OuterVolumeSpecName: "kube-api-access-gj9ss") pod "f2fc98c7-d83f-488f-9cef-e72263b14ae1" (UID: "f2fc98c7-d83f-488f-9cef-e72263b14ae1"). InnerVolumeSpecName "kube-api-access-gj9ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:57.991146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f0c8a1-7172-440d-b534-75dfbb74be47-plugins-config-volume\") pod \"08f0c8a1-7172-440d-b534-75dfbb74be47\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " Jul 29 07:43:57.991304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991186 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spspb\" (UniqueName: \"kubernetes.io/projected/08f0c8a1-7172-440d-b534-75dfbb74be47-kube-api-access-spspb\") pod \"08f0c8a1-7172-440d-b534-75dfbb74be47\" (UID: \"08f0c8a1-7172-440d-b534-75dfbb74be47\") " Jul 29 07:43:57.991522 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991493 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2fc98c7-d83f-488f-9cef-e72263b14ae1-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:57.991522 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991521 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca45cb-e628-4b6d-a406-b375f2cb63eb-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:57.991684 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991537 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9ss\" (UniqueName: \"kubernetes.io/projected/f2fc98c7-d83f-488f-9cef-e72263b14ae1-kube-api-access-gj9ss\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:57.991684 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991552 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8nhm\" (UniqueName: \"kubernetes.io/projected/52ca45cb-e628-4b6d-a406-b375f2cb63eb-kube-api-access-p8nhm\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:57.991684 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.991595 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f0c8a1-7172-440d-b534-75dfbb74be47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08f0c8a1-7172-440d-b534-75dfbb74be47" (UID: "08f0c8a1-7172-440d-b534-75dfbb74be47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:43:57.993089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:57.993072 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f0c8a1-7172-440d-b534-75dfbb74be47-kube-api-access-spspb" (OuterVolumeSpecName: "kube-api-access-spspb") pod "08f0c8a1-7172-440d-b534-75dfbb74be47" (UID: "08f0c8a1-7172-440d-b534-75dfbb74be47"). InnerVolumeSpecName "kube-api-access-spspb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:43:58.092173 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.092146 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spspb\" (UniqueName: \"kubernetes.io/projected/08f0c8a1-7172-440d-b534-75dfbb74be47-kube-api-access-spspb\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:58.092173 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.092170 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08f0c8a1-7172-440d-b534-75dfbb74be47-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:43:58.657090 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.657059 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" Jul 29 07:43:58.657543 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.657063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9" event={"ID":"f2fc98c7-d83f-488f-9cef-e72263b14ae1","Type":"ContainerDied","Data":"432e5f0c0f4b1e88fb4c5477fa78e393910e34ff4671a058c4ef7aae0e920d20"} Jul 29 07:43:58.657543 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.657208 2565 scope.go:117] "RemoveContainer" containerID="91995636a293cbc397b5aae79d1b15d4f706ca8757dee320231dd0da88d28548" Jul 29 07:43:58.658532 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.658502 2565 generic.go:356] "Generic (PLEG): container finished" podID="08f0c8a1-7172-440d-b534-75dfbb74be47" containerID="8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec" exitCode=0 Jul 29 07:43:58.658605 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.658554 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" Jul 29 07:43:58.658666 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.658593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" event={"ID":"08f0c8a1-7172-440d-b534-75dfbb74be47","Type":"ContainerDied","Data":"8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec"} Jul 29 07:43:58.658666 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.658646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9" event={"ID":"08f0c8a1-7172-440d-b534-75dfbb74be47","Type":"ContainerDied","Data":"810d018b78dfe5b3aa1278448317a1603f2dc9c3301ca0323404e590db485c0b"} Jul 29 07:43:58.660156 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.660136 2565 generic.go:356] "Generic (PLEG): container finished" podID="52ca45cb-e628-4b6d-a406-b375f2cb63eb" containerID="59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21" exitCode=0 Jul 29 07:43:58.660276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.660181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" event={"ID":"52ca45cb-e628-4b6d-a406-b375f2cb63eb","Type":"ContainerDied","Data":"59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21"} Jul 29 07:43:58.660276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.660219 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" event={"ID":"52ca45cb-e628-4b6d-a406-b375f2cb63eb","Type":"ContainerDied","Data":"536cdff8a815cb721fe522484ca409d1f87e8bd31886a8a77dbbdbce1608acd7"} Jul 29 07:43:58.660276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.660186 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8" Jul 29 07:43:58.668068 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.668051 2565 scope.go:117] "RemoveContainer" containerID="8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec" Jul 29 07:43:58.676570 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.676552 2565 scope.go:117] "RemoveContainer" containerID="8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec" Jul 29 07:43:58.676831 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:43:58.676813 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec\": container with ID starting with 8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec not found: ID does not exist" containerID="8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec" Jul 29 07:43:58.676874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.676840 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec"} err="failed to get container status \"8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec\": rpc error: code = NotFound desc = could not find container \"8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec\": container with ID starting with 8cdda6b53a9a06dacde7ea3c75250cf238544eec87c361827084587dfd6804ec not found: ID does not exist" Jul 29 07:43:58.676874 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.676857 2565 scope.go:117] "RemoveContainer" containerID="59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21" Jul 29 07:43:58.684227 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.684210 2565 scope.go:117] "RemoveContainer" containerID="59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21" Jul 29 07:43:58.684457 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:43:58.684444 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21\": container with ID starting with 59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21 not found: ID does not exist" containerID="59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21" Jul 29 07:43:58.684495 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.684463 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21"} err="failed to get container status \"59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21\": rpc error: code = NotFound desc = could not find container \"59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21\": container with ID starting with 59913c5f9bc5a374c3edfa563cbf9a177398d967145340182758e40a7f64fa21 not found: ID does not exist" Jul 29 07:43:58.705549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.705522 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9"] Jul 29 07:43:58.708637 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.708616 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d04dd228-67f57df474-595k9"] Jul 29 07:43:58.731264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.731240 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8"] Jul 29 07:43:58.738633 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.738610 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d04dd228-97b8c5556-bzlh8"] Jul 29 07:43:58.760049 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.759979 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9"] Jul 29 07:43:58.766878 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:58.766851 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d04dd228-75c774794d-wg6z9"] Jul 29 07:43:59.019942 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:59.019846 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f0c8a1-7172-440d-b534-75dfbb74be47" path="/var/lib/kubelet/pods/08f0c8a1-7172-440d-b534-75dfbb74be47/volumes" Jul 29 07:43:59.020562 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:59.020534 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ca45cb-e628-4b6d-a406-b375f2cb63eb" path="/var/lib/kubelet/pods/52ca45cb-e628-4b6d-a406-b375f2cb63eb/volumes" Jul 29 07:43:59.020944 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:43:59.020929 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fc98c7-d83f-488f-9cef-e72263b14ae1" path="/var/lib/kubelet/pods/f2fc98c7-d83f-488f-9cef-e72263b14ae1/volumes" Jul 29 07:44:03.060963 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.060927 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899"] Jul 29 07:44:03.061329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061282 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ca45cb-e628-4b6d-a406-b375f2cb63eb" containerName="payload-processing" Jul 29 07:44:03.061329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061294 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ca45cb-e628-4b6d-a406-b375f2cb63eb" containerName="payload-processing" Jul 29 07:44:03.061329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061315 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fc98c7-d83f-488f-9cef-e72263b14ae1" containerName="maas-api" Jul 29 07:44:03.061329 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061320 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fc98c7-d83f-488f-9cef-e72263b14ae1" containerName="maas-api" Jul 29 07:44:03.061462 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061334 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f0c8a1-7172-440d-b534-75dfbb74be47" containerName="payload-pre-processing" Jul 29 07:44:03.061462 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061340 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f0c8a1-7172-440d-b534-75dfbb74be47" containerName="payload-pre-processing" Jul 29 07:44:03.061462 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061390 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f0c8a1-7172-440d-b534-75dfbb74be47" containerName="payload-pre-processing" Jul 29 07:44:03.061462 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061400 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ca45cb-e628-4b6d-a406-b375f2cb63eb" containerName="payload-processing" Jul 29 07:44:03.061462 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.061406 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fc98c7-d83f-488f-9cef-e72263b14ae1" containerName="maas-api" Jul 29 07:44:03.066133 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.066115 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.071156 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.071128 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:44:03.072930 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.072609 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:44:03.072930 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.072626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3564323f-serving-cert\"" Jul 29 07:44:03.072930 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.072642 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:44:03.098539 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.098512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899"] Jul 29 07:44:03.138052 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.138011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.138248 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.138181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8q5\" (UniqueName: \"kubernetes.io/projected/6274e8d9-9b32-402a-867f-ee377d3555d8-kube-api-access-fd8q5\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.138248 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.138241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6274e8d9-9b32-402a-867f-ee377d3555d8-openshift-service-ca\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.239249 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.239211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.239430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.239270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8q5\" (UniqueName: \"kubernetes.io/projected/6274e8d9-9b32-402a-867f-ee377d3555d8-kube-api-access-fd8q5\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.239430 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.239305 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6274e8d9-9b32-402a-867f-ee377d3555d8-openshift-service-ca\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.239430 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:44:03.239364 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-serving-cert: secret "maas-api-e2e-derive-3564323f-serving-cert" not found Jul 29 07:44:03.239561 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:44:03.239432 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls podName:6274e8d9-9b32-402a-867f-ee377d3555d8 nodeName:}" failed. No retries permitted until 2026-07-29 07:44:03.739413035 +0000 UTC m=+1725.345591124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls") pod "maas-api-e2e-derive-3564323f-759cb9484f-cm899" (UID: "6274e8d9-9b32-402a-867f-ee377d3555d8") : secret "maas-api-e2e-derive-3564323f-serving-cert" not found Jul 29 07:44:03.239975 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.239940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6274e8d9-9b32-402a-867f-ee377d3555d8-openshift-service-ca\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.257229 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.257204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8q5\" (UniqueName: \"kubernetes.io/projected/6274e8d9-9b32-402a-867f-ee377d3555d8-kube-api-access-fd8q5\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.744133 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.744101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.746484 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.746456 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls\") pod \"maas-api-e2e-derive-3564323f-759cb9484f-cm899\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:03.976502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:03.976467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:04.235252 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:04.235220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899"] Jul 29 07:44:04.241815 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:44:04.241790 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6274e8d9_9b32_402a_867f_ee377d3555d8.slice/crio-5f2acd3ad032c36c9c18e59380180360f408b3fbebd2b0ad9204114a4baea4fd WatchSource:0}: Error finding container 5f2acd3ad032c36c9c18e59380180360f408b3fbebd2b0ad9204114a4baea4fd: Status 404 returned error can't find the container with id 5f2acd3ad032c36c9c18e59380180360f408b3fbebd2b0ad9204114a4baea4fd Jul 29 07:44:04.699209 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:04.699170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" event={"ID":"6274e8d9-9b32-402a-867f-ee377d3555d8","Type":"ContainerStarted","Data":"e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085"} Jul 29 07:44:04.699379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:04.699263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" event={"ID":"6274e8d9-9b32-402a-867f-ee377d3555d8","Type":"ContainerStarted","Data":"5f2acd3ad032c36c9c18e59380180360f408b3fbebd2b0ad9204114a4baea4fd"} Jul 29 07:44:04.699379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:04.699313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:04.807528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:04.807473 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" podStartSLOduration=1.518875119 podStartE2EDuration="1.80745479s" totalImagesPullingTime="288.579671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:44:04.798402323 +0000 UTC m=+1726.404580433" watchObservedRunningTime="2026-07-29 07:44:04.80745479 +0000 UTC m=+1726.413633097" Jul 29 07:44:05.067529 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.067443 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5"] Jul 29 07:44:05.070997 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.070975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.078034 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.078011 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3564323f\"" Jul 29 07:44:05.078995 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.078979 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3564323f-dockercfg-hvdfx\"" Jul 29 07:44:05.093331 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.093301 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw"] Jul 29 07:44:05.096766 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.096746 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.112091 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.112068 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5"] Jul 29 07:44:05.129284 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.129246 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw"] Jul 29 07:44:05.154334 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.154302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba75c7d5-c62a-4730-b401-3813db0758b1-plugins-config-volume\") pod \"payload-processing-e2e-derive-3564323f-675df77b98-9xfv5\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.154479 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.154338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96nf\" (UniqueName: \"kubernetes.io/projected/ba75c7d5-c62a-4730-b401-3813db0758b1-kube-api-access-v96nf\") pod \"payload-processing-e2e-derive-3564323f-675df77b98-9xfv5\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.154479 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.154357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkh98\" (UniqueName: \"kubernetes.io/projected/32effa40-a016-4cb7-8a23-aaac8e446862-kube-api-access-nkh98\") pod \"payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.154479 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.154393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32effa40-a016-4cb7-8a23-aaac8e446862-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.255972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.255922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba75c7d5-c62a-4730-b401-3813db0758b1-plugins-config-volume\") pod \"payload-processing-e2e-derive-3564323f-675df77b98-9xfv5\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.255972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.255972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96nf\" (UniqueName: \"kubernetes.io/projected/ba75c7d5-c62a-4730-b401-3813db0758b1-kube-api-access-v96nf\") pod \"payload-processing-e2e-derive-3564323f-675df77b98-9xfv5\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.256433 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.255991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkh98\" (UniqueName: \"kubernetes.io/projected/32effa40-a016-4cb7-8a23-aaac8e446862-kube-api-access-nkh98\") pod \"payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.256433 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.256015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32effa40-a016-4cb7-8a23-aaac8e446862-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.256620 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.256600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba75c7d5-c62a-4730-b401-3813db0758b1-plugins-config-volume\") pod \"payload-processing-e2e-derive-3564323f-675df77b98-9xfv5\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.256661 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.256620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32effa40-a016-4cb7-8a23-aaac8e446862-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.272448 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.272419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96nf\" (UniqueName: \"kubernetes.io/projected/ba75c7d5-c62a-4730-b401-3813db0758b1-kube-api-access-v96nf\") pod \"payload-processing-e2e-derive-3564323f-675df77b98-9xfv5\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.272599 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.272577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkh98\" (UniqueName: \"kubernetes.io/projected/32effa40-a016-4cb7-8a23-aaac8e446862-kube-api-access-nkh98\") pod \"payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.381539 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.381478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.406244 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.406205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:05.554402 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:44:05.554372 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba75c7d5_c62a_4730_b401_3813db0758b1.slice/crio-6d47e919ebe6ee7e8ad983bb8010b6cd9dbdc63f83fe5b4c2a3349ba689fb446 WatchSource:0}: Error finding container 6d47e919ebe6ee7e8ad983bb8010b6cd9dbdc63f83fe5b4c2a3349ba689fb446: Status 404 returned error can't find the container with id 6d47e919ebe6ee7e8ad983bb8010b6cd9dbdc63f83fe5b4c2a3349ba689fb446 Jul 29 07:44:05.554841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.554817 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5"] Jul 29 07:44:05.706264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.706184 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" event={"ID":"ba75c7d5-c62a-4730-b401-3813db0758b1","Type":"ContainerStarted","Data":"13ac150a37c04e3e821dccddee0c5eede9164980e1370a6bc4fe8428798abb2f"} Jul 29 07:44:05.706264 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.706236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" event={"ID":"ba75c7d5-c62a-4730-b401-3813db0758b1","Type":"ContainerStarted","Data":"6d47e919ebe6ee7e8ad983bb8010b6cd9dbdc63f83fe5b4c2a3349ba689fb446"} Jul 29 07:44:05.706469 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.706451 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:05.729028 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.728985 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" podStartSLOduration=0.728970856 podStartE2EDuration="728.970856ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:44:05.727232058 +0000 UTC m=+1727.333410165" watchObservedRunningTime="2026-07-29 07:44:05.728970856 +0000 UTC m=+1727.335148955" Jul 29 07:44:05.764395 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:05.764367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw"] Jul 29 07:44:05.764987 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:44:05.764936 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32effa40_a016_4cb7_8a23_aaac8e446862.slice/crio-15d55253d368d7f66e695b914c734d9491d482c0267a7ab3044f724825b8f5ee WatchSource:0}: Error finding container 15d55253d368d7f66e695b914c734d9491d482c0267a7ab3044f724825b8f5ee: Status 404 returned error can't find the container with id 15d55253d368d7f66e695b914c734d9491d482c0267a7ab3044f724825b8f5ee Jul 29 07:44:06.712766 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:06.712731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" event={"ID":"32effa40-a016-4cb7-8a23-aaac8e446862","Type":"ContainerStarted","Data":"9bd0e4d51771c8f3ea195883ee782c26e5e359465dd954ebb54a33a74a95a02b"} Jul 29 07:44:06.713186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:06.712791 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:06.713186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:06.712811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" event={"ID":"32effa40-a016-4cb7-8a23-aaac8e446862","Type":"ContainerStarted","Data":"15d55253d368d7f66e695b914c734d9491d482c0267a7ab3044f724825b8f5ee"} Jul 29 07:44:06.746250 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:06.746202 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" podStartSLOduration=1.746187773 podStartE2EDuration="1.746187773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:44:06.74347872 +0000 UTC m=+1728.349656827" watchObservedRunningTime="2026-07-29 07:44:06.746187773 +0000 UTC m=+1728.352365880" Jul 29 07:44:10.711558 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:10.711528 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:16.714720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:16.714685 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:17.720884 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:17.720853 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:33.520324 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.520290 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899"] Jul 29 07:44:33.520735 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.520512 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" podUID="6274e8d9-9b32-402a-867f-ee377d3555d8" containerName="maas-api" containerID="cri-o://e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085" gracePeriod=30 Jul 29 07:44:33.614980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.614000 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5"] Jul 29 07:44:33.614980 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.614246 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" podUID="ba75c7d5-c62a-4730-b401-3813db0758b1" containerName="payload-processing" containerID="cri-o://13ac150a37c04e3e821dccddee0c5eede9164980e1370a6bc4fe8428798abb2f" gracePeriod=30 Jul 29 07:44:33.628052 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.628021 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw"] Jul 29 07:44:33.628310 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.628253 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" podUID="32effa40-a016-4cb7-8a23-aaac8e446862" containerName="payload-pre-processing" containerID="cri-o://9bd0e4d51771c8f3ea195883ee782c26e5e359465dd954ebb54a33a74a95a02b" gracePeriod=30 Jul 29 07:44:33.788730 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.788640 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:33.865568 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.865533 2565 generic.go:356] "Generic (PLEG): container finished" podID="ba75c7d5-c62a-4730-b401-3813db0758b1" containerID="13ac150a37c04e3e821dccddee0c5eede9164980e1370a6bc4fe8428798abb2f" exitCode=0 Jul 29 07:44:33.865722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.865604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" event={"ID":"ba75c7d5-c62a-4730-b401-3813db0758b1","Type":"ContainerDied","Data":"13ac150a37c04e3e821dccddee0c5eede9164980e1370a6bc4fe8428798abb2f"} Jul 29 07:44:33.868122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.868072 2565 generic.go:356] "Generic (PLEG): container finished" podID="6274e8d9-9b32-402a-867f-ee377d3555d8" containerID="e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085" exitCode=0 Jul 29 07:44:33.868246 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.868207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" event={"ID":"6274e8d9-9b32-402a-867f-ee377d3555d8","Type":"ContainerDied","Data":"e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085"} Jul 29 07:44:33.868330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.868258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" event={"ID":"6274e8d9-9b32-402a-867f-ee377d3555d8","Type":"ContainerDied","Data":"5f2acd3ad032c36c9c18e59380180360f408b3fbebd2b0ad9204114a4baea4fd"} Jul 29 07:44:33.868330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.868303 2565 scope.go:117] "RemoveContainer" containerID="e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085" Jul 29 07:44:33.868459 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.868430 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899" Jul 29 07:44:33.872497 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.872475 2565 generic.go:356] "Generic (PLEG): container finished" podID="32effa40-a016-4cb7-8a23-aaac8e446862" containerID="9bd0e4d51771c8f3ea195883ee782c26e5e359465dd954ebb54a33a74a95a02b" exitCode=0 Jul 29 07:44:33.872618 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.872572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" event={"ID":"32effa40-a016-4cb7-8a23-aaac8e446862","Type":"ContainerDied","Data":"9bd0e4d51771c8f3ea195883ee782c26e5e359465dd954ebb54a33a74a95a02b"} Jul 29 07:44:33.883742 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.883666 2565 scope.go:117] "RemoveContainer" containerID="e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085" Jul 29 07:44:33.884050 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:44:33.884022 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085\": container with ID starting with e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085 not found: ID does not exist" containerID="e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085" Jul 29 07:44:33.884153 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.884062 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085"} err="failed to get container status \"e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085\": rpc error: code = NotFound desc = could not find container \"e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085\": container with ID starting with e6a24ddeb2329dbe90da2a72c456cc18b8c9989b8ca62b375aada15d855a7085 not found: ID does not exist" Jul 29 07:44:33.901022 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.900800 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd8q5\" (UniqueName: \"kubernetes.io/projected/6274e8d9-9b32-402a-867f-ee377d3555d8-kube-api-access-fd8q5\") pod \"6274e8d9-9b32-402a-867f-ee377d3555d8\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " Jul 29 07:44:33.901022 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.900852 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6274e8d9-9b32-402a-867f-ee377d3555d8-openshift-service-ca\") pod \"6274e8d9-9b32-402a-867f-ee377d3555d8\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " Jul 29 07:44:33.901022 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.900980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls\") pod \"6274e8d9-9b32-402a-867f-ee377d3555d8\" (UID: \"6274e8d9-9b32-402a-867f-ee377d3555d8\") " Jul 29 07:44:33.901938 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.901911 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6274e8d9-9b32-402a-867f-ee377d3555d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6274e8d9-9b32-402a-867f-ee377d3555d8" (UID: "6274e8d9-9b32-402a-867f-ee377d3555d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:44:33.904179 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.904152 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6274e8d9-9b32-402a-867f-ee377d3555d8-kube-api-access-fd8q5" (OuterVolumeSpecName: "kube-api-access-fd8q5") pod "6274e8d9-9b32-402a-867f-ee377d3555d8" (UID: "6274e8d9-9b32-402a-867f-ee377d3555d8"). InnerVolumeSpecName "kube-api-access-fd8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:33.904287 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.904203 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6274e8d9-9b32-402a-867f-ee377d3555d8" (UID: "6274e8d9-9b32-402a-867f-ee377d3555d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:44:33.907357 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.907335 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:33.913669 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:33.913651 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:34.002440 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002404 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkh98\" (UniqueName: \"kubernetes.io/projected/32effa40-a016-4cb7-8a23-aaac8e446862-kube-api-access-nkh98\") pod \"32effa40-a016-4cb7-8a23-aaac8e446862\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " Jul 29 07:44:34.002599 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002498 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32effa40-a016-4cb7-8a23-aaac8e446862-plugins-config-volume\") pod \"32effa40-a016-4cb7-8a23-aaac8e446862\" (UID: \"32effa40-a016-4cb7-8a23-aaac8e446862\") " Jul 29 07:44:34.002599 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v96nf\" (UniqueName: \"kubernetes.io/projected/ba75c7d5-c62a-4730-b401-3813db0758b1-kube-api-access-v96nf\") pod \"ba75c7d5-c62a-4730-b401-3813db0758b1\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " Jul 29 07:44:34.002599 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba75c7d5-c62a-4730-b401-3813db0758b1-plugins-config-volume\") pod \"ba75c7d5-c62a-4730-b401-3813db0758b1\" (UID: \"ba75c7d5-c62a-4730-b401-3813db0758b1\") " Jul 29 07:44:34.002870 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002851 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd8q5\" (UniqueName: \"kubernetes.io/projected/6274e8d9-9b32-402a-867f-ee377d3555d8-kube-api-access-fd8q5\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.002946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002875 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6274e8d9-9b32-402a-867f-ee377d3555d8-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.002946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002880 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32effa40-a016-4cb7-8a23-aaac8e446862-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32effa40-a016-4cb7-8a23-aaac8e446862" (UID: "32effa40-a016-4cb7-8a23-aaac8e446862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:44:34.002946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002889 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6274e8d9-9b32-402a-867f-ee377d3555d8-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.002946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.002929 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba75c7d5-c62a-4730-b401-3813db0758b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba75c7d5-c62a-4730-b401-3813db0758b1" (UID: "ba75c7d5-c62a-4730-b401-3813db0758b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:44:34.005011 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.004989 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba75c7d5-c62a-4730-b401-3813db0758b1-kube-api-access-v96nf" (OuterVolumeSpecName: "kube-api-access-v96nf") pod "ba75c7d5-c62a-4730-b401-3813db0758b1" (UID: "ba75c7d5-c62a-4730-b401-3813db0758b1"). InnerVolumeSpecName "kube-api-access-v96nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:34.005130 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.005015 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32effa40-a016-4cb7-8a23-aaac8e446862-kube-api-access-nkh98" (OuterVolumeSpecName: "kube-api-access-nkh98") pod "32effa40-a016-4cb7-8a23-aaac8e446862" (UID: "32effa40-a016-4cb7-8a23-aaac8e446862"). InnerVolumeSpecName "kube-api-access-nkh98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:44:34.103651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.103616 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32effa40-a016-4cb7-8a23-aaac8e446862-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.103651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.103645 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96nf\" (UniqueName: \"kubernetes.io/projected/ba75c7d5-c62a-4730-b401-3813db0758b1-kube-api-access-v96nf\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.103651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.103657 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba75c7d5-c62a-4730-b401-3813db0758b1-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.103884 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.103669 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkh98\" (UniqueName: \"kubernetes.io/projected/32effa40-a016-4cb7-8a23-aaac8e446862-kube-api-access-nkh98\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:44:34.201082 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.201053 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899"] Jul 29 07:44:34.203134 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.203113 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3564323f-759cb9484f-cm899"] Jul 29 07:44:34.879514 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.879486 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" Jul 29 07:44:34.880078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.879534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw" event={"ID":"32effa40-a016-4cb7-8a23-aaac8e446862","Type":"ContainerDied","Data":"15d55253d368d7f66e695b914c734d9491d482c0267a7ab3044f724825b8f5ee"} Jul 29 07:44:34.880078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.879596 2565 scope.go:117] "RemoveContainer" containerID="9bd0e4d51771c8f3ea195883ee782c26e5e359465dd954ebb54a33a74a95a02b" Jul 29 07:44:34.880849 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.880831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" event={"ID":"ba75c7d5-c62a-4730-b401-3813db0758b1","Type":"ContainerDied","Data":"6d47e919ebe6ee7e8ad983bb8010b6cd9dbdc63f83fe5b4c2a3349ba689fb446"} Jul 29 07:44:34.880849 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.880840 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5" Jul 29 07:44:34.889547 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.889532 2565 scope.go:117] "RemoveContainer" containerID="13ac150a37c04e3e821dccddee0c5eede9164980e1370a6bc4fe8428798abb2f" Jul 29 07:44:34.912015 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.911992 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw"] Jul 29 07:44:34.918096 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.918074 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3564323f-658b58b6cb-2zfdw"] Jul 29 07:44:34.934485 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.934461 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5"] Jul 29 07:44:34.944520 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:34.944501 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3564323f-675df77b98-9xfv5"] Jul 29 07:44:35.019335 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:35.019313 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32effa40-a016-4cb7-8a23-aaac8e446862" path="/var/lib/kubelet/pods/32effa40-a016-4cb7-8a23-aaac8e446862/volumes" Jul 29 07:44:35.019695 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:35.019682 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6274e8d9-9b32-402a-867f-ee377d3555d8" path="/var/lib/kubelet/pods/6274e8d9-9b32-402a-867f-ee377d3555d8/volumes" Jul 29 07:44:35.020018 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:44:35.020006 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba75c7d5-c62a-4730-b401-3813db0758b1" path="/var/lib/kubelet/pods/ba75c7d5-c62a-4730-b401-3813db0758b1/volumes" Jul 29 07:46:22.352075 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352043 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l"] Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352391 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32effa40-a016-4cb7-8a23-aaac8e446862" containerName="payload-pre-processing" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352402 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32effa40-a016-4cb7-8a23-aaac8e446862" containerName="payload-pre-processing" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352412 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6274e8d9-9b32-402a-867f-ee377d3555d8" containerName="maas-api" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352418 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6274e8d9-9b32-402a-867f-ee377d3555d8" containerName="maas-api" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352451 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba75c7d5-c62a-4730-b401-3813db0758b1" containerName="payload-processing" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352457 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba75c7d5-c62a-4730-b401-3813db0758b1" containerName="payload-processing" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352506 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32effa40-a016-4cb7-8a23-aaac8e446862" containerName="payload-pre-processing" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352513 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba75c7d5-c62a-4730-b401-3813db0758b1" containerName="payload-processing" Jul 29 07:46:22.352553 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.352521 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6274e8d9-9b32-402a-867f-ee377d3555d8" containerName="maas-api" Jul 29 07:46:22.355489 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.355471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.358384 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.358361 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2c7c574-serving-cert\"" Jul 29 07:46:22.358575 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.358556 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:46:22.358685 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.358668 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:46:22.360001 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.359983 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:46:22.368324 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.368302 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l"] Jul 29 07:46:22.494278 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.494245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.494438 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.494292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztwb\" (UniqueName: \"kubernetes.io/projected/7ae20ece-653f-474b-ba28-202f89e94b2a-kube-api-access-fztwb\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.494438 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.494380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae20ece-653f-474b-ba28-202f89e94b2a-openshift-service-ca\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.595482 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.595454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae20ece-653f-474b-ba28-202f89e94b2a-openshift-service-ca\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.595633 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.595538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.595633 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.595586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztwb\" (UniqueName: \"kubernetes.io/projected/7ae20ece-653f-474b-ba28-202f89e94b2a-kube-api-access-fztwb\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.595714 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:46:22.595689 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-serving-cert: secret "maas-api-e2e-mt-e2c7c574-serving-cert" not found Jul 29 07:46:22.595775 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:46:22.595763 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls podName:7ae20ece-653f-474b-ba28-202f89e94b2a nodeName:}" failed. No retries permitted until 2026-07-29 07:46:23.095742898 +0000 UTC m=+1864.701920984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls") pod "maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" (UID: "7ae20ece-653f-474b-ba28-202f89e94b2a") : secret "maas-api-e2e-mt-e2c7c574-serving-cert" not found Jul 29 07:46:22.596213 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.596195 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae20ece-653f-474b-ba28-202f89e94b2a-openshift-service-ca\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:22.608722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:22.608654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztwb\" (UniqueName: \"kubernetes.io/projected/7ae20ece-653f-474b-ba28-202f89e94b2a-kube-api-access-fztwb\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:23.100193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.100162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:23.102407 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.102377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls\") pod \"maas-api-e2e-mt-e2c7c574-6888694b7-cm68l\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:23.262358 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.262327 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc"] Jul 29 07:46:23.267624 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.266550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.267774 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.267739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:23.272443 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.272420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e2c7c574-dockercfg-d5gzq\"" Jul 29 07:46:23.273063 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.272762 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e2c7c574\"" Jul 29 07:46:23.289659 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.289343 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc"] Jul 29 07:46:23.311413 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.311382 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c"] Jul 29 07:46:23.315304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.315278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.332211 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.332138 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c"] Jul 29 07:46:23.402804 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.402769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swhs\" (UniqueName: \"kubernetes.io/projected/511dcbe5-2703-4084-a081-a87ee369a40a-kube-api-access-2swhs\") pod \"payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.403233 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.402917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511dcbe5-2703-4084-a081-a87ee369a40a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.403233 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.403066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5db11ac-6f16-4272-9917-404552ca181d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.403233 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.403106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbd8\" (UniqueName: \"kubernetes.io/projected/f5db11ac-6f16-4272-9917-404552ca181d-kube-api-access-wdbd8\") pod \"payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.431616 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.431578 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l"] Jul 29 07:46:23.432500 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:46:23.432466 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae20ece_653f_474b_ba28_202f89e94b2a.slice/crio-103cbddf89459864c84a17be09786c62000abee81a5eb8b2811b239a6842cb1d WatchSource:0}: Error finding container 103cbddf89459864c84a17be09786c62000abee81a5eb8b2811b239a6842cb1d: Status 404 returned error can't find the container with id 103cbddf89459864c84a17be09786c62000abee81a5eb8b2811b239a6842cb1d Jul 29 07:46:23.503981 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.503935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5db11ac-6f16-4272-9917-404552ca181d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.504076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.503992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbd8\" (UniqueName: \"kubernetes.io/projected/f5db11ac-6f16-4272-9917-404552ca181d-kube-api-access-wdbd8\") pod \"payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.504076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.504021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swhs\" (UniqueName: \"kubernetes.io/projected/511dcbe5-2703-4084-a081-a87ee369a40a-kube-api-access-2swhs\") pod \"payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.504076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.504064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511dcbe5-2703-4084-a081-a87ee369a40a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.504660 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.504642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5db11ac-6f16-4272-9917-404552ca181d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.504778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.504759 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511dcbe5-2703-4084-a081-a87ee369a40a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.513783 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.513753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swhs\" (UniqueName: \"kubernetes.io/projected/511dcbe5-2703-4084-a081-a87ee369a40a-kube-api-access-2swhs\") pod \"payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.514849 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.514828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbd8\" (UniqueName: \"kubernetes.io/projected/f5db11ac-6f16-4272-9917-404552ca181d-kube-api-access-wdbd8\") pod \"payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.579046 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.579025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:23.637072 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.637039 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:23.725179 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.725150 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc"] Jul 29 07:46:23.726577 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:46:23.726547 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5db11ac_6f16_4272_9917_404552ca181d.slice/crio-812e2cd00b5ac007a8d9030bc9ff8163d04cdd5e606c8d1f7c94b299fe8e658a WatchSource:0}: Error finding container 812e2cd00b5ac007a8d9030bc9ff8163d04cdd5e606c8d1f7c94b299fe8e658a: Status 404 returned error can't find the container with id 812e2cd00b5ac007a8d9030bc9ff8163d04cdd5e606c8d1f7c94b299fe8e658a Jul 29 07:46:23.787544 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:46:23.787518 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511dcbe5_2703_4084_a081_a87ee369a40a.slice/crio-d81b19683151613ace66178f731328adebda5165a494b606277aef650dd06fb5 WatchSource:0}: Error finding container d81b19683151613ace66178f731328adebda5165a494b606277aef650dd06fb5: Status 404 returned error can't find the container with id d81b19683151613ace66178f731328adebda5165a494b606277aef650dd06fb5 Jul 29 07:46:23.787751 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:23.787723 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c"] Jul 29 07:46:24.441043 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.441006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" event={"ID":"511dcbe5-2703-4084-a081-a87ee369a40a","Type":"ContainerStarted","Data":"fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa"} Jul 29 07:46:24.441512 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.441069 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:46:24.441512 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.441094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" event={"ID":"511dcbe5-2703-4084-a081-a87ee369a40a","Type":"ContainerStarted","Data":"d81b19683151613ace66178f731328adebda5165a494b606277aef650dd06fb5"} Jul 29 07:46:24.442434 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.442410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" event={"ID":"f5db11ac-6f16-4272-9917-404552ca181d","Type":"ContainerStarted","Data":"cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf"} Jul 29 07:46:24.442565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.442458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" event={"ID":"f5db11ac-6f16-4272-9917-404552ca181d","Type":"ContainerStarted","Data":"812e2cd00b5ac007a8d9030bc9ff8163d04cdd5e606c8d1f7c94b299fe8e658a"} Jul 29 07:46:24.442565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.442485 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:24.443711 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.443685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" event={"ID":"7ae20ece-653f-474b-ba28-202f89e94b2a","Type":"ContainerStarted","Data":"61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda"} Jul 29 07:46:24.443805 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.443723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" event={"ID":"7ae20ece-653f-474b-ba28-202f89e94b2a","Type":"ContainerStarted","Data":"103cbddf89459864c84a17be09786c62000abee81a5eb8b2811b239a6842cb1d"} Jul 29 07:46:24.443805 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.443754 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:24.493461 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.493420 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" podStartSLOduration=1.49340795 podStartE2EDuration="1.49340795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:46:24.492342025 +0000 UTC m=+1866.098520125" watchObservedRunningTime="2026-07-29 07:46:24.49340795 +0000 UTC m=+1866.099586057" Jul 29 07:46:24.493546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.493489 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" podStartSLOduration=1.493486181 podStartE2EDuration="1.493486181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:46:24.467687506 +0000 UTC m=+1866.073865615" watchObservedRunningTime="2026-07-29 07:46:24.493486181 +0000 UTC m=+1866.099664334" Jul 29 07:46:24.519359 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:24.519324 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" podStartSLOduration=1.9922223730000002 podStartE2EDuration="2.519312678s" totalImagesPullingTime="527.090305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:46:24.51773697 +0000 UTC m=+1866.123915078" watchObservedRunningTime="2026-07-29 07:46:24.519312678 +0000 UTC m=+1866.125490824" Jul 29 07:46:30.455845 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:30.455815 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:46:35.451901 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:35.451867 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:46:35.452352 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:46:35.451977 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:47:05.926549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:05.926516 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l"] Jul 29 07:47:05.927075 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:05.926724 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" podUID="7ae20ece-653f-474b-ba28-202f89e94b2a" containerName="maas-api" containerID="cri-o://61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda" gracePeriod=30 Jul 29 07:47:05.980189 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:05.980158 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c"] Jul 29 07:47:05.980387 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:05.980338 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" podUID="511dcbe5-2703-4084-a081-a87ee369a40a" containerName="payload-processing" containerID="cri-o://fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa" gracePeriod=30 Jul 29 07:47:05.993151 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:05.993124 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc"] Jul 29 07:47:05.993374 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:05.993341 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" podUID="f5db11ac-6f16-4272-9917-404552ca181d" containerName="payload-pre-processing" containerID="cri-o://cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf" gracePeriod=30 Jul 29 07:47:06.269768 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.269742 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:47:06.287582 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.287565 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:47:06.292272 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.292253 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:47:06.359109 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359076 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fztwb\" (UniqueName: \"kubernetes.io/projected/7ae20ece-653f-474b-ba28-202f89e94b2a-kube-api-access-fztwb\") pod \"7ae20ece-653f-474b-ba28-202f89e94b2a\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " Jul 29 07:47:06.359262 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359133 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae20ece-653f-474b-ba28-202f89e94b2a-openshift-service-ca\") pod \"7ae20ece-653f-474b-ba28-202f89e94b2a\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " Jul 29 07:47:06.359262 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359171 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls\") pod \"7ae20ece-653f-474b-ba28-202f89e94b2a\" (UID: \"7ae20ece-653f-474b-ba28-202f89e94b2a\") " Jul 29 07:47:06.359262 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359198 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2swhs\" (UniqueName: \"kubernetes.io/projected/511dcbe5-2703-4084-a081-a87ee369a40a-kube-api-access-2swhs\") pod \"511dcbe5-2703-4084-a081-a87ee369a40a\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " Jul 29 07:47:06.359436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359271 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511dcbe5-2703-4084-a081-a87ee369a40a-plugins-config-volume\") pod \"511dcbe5-2703-4084-a081-a87ee369a40a\" (UID: \"511dcbe5-2703-4084-a081-a87ee369a40a\") " Jul 29 07:47:06.359436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359319 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5db11ac-6f16-4272-9917-404552ca181d-plugins-config-volume\") pod \"f5db11ac-6f16-4272-9917-404552ca181d\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " Jul 29 07:47:06.359436 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdbd8\" (UniqueName: \"kubernetes.io/projected/f5db11ac-6f16-4272-9917-404552ca181d-kube-api-access-wdbd8\") pod \"f5db11ac-6f16-4272-9917-404552ca181d\" (UID: \"f5db11ac-6f16-4272-9917-404552ca181d\") " Jul 29 07:47:06.359674 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359642 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae20ece-653f-474b-ba28-202f89e94b2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ae20ece-653f-474b-ba28-202f89e94b2a" (UID: "7ae20ece-653f-474b-ba28-202f89e94b2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:47:06.359815 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.359788 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511dcbe5-2703-4084-a081-a87ee369a40a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "511dcbe5-2703-4084-a081-a87ee369a40a" (UID: "511dcbe5-2703-4084-a081-a87ee369a40a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:47:06.360083 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.360055 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5db11ac-6f16-4272-9917-404552ca181d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5db11ac-6f16-4272-9917-404552ca181d" (UID: "f5db11ac-6f16-4272-9917-404552ca181d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:47:06.361404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.361377 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511dcbe5-2703-4084-a081-a87ee369a40a-kube-api-access-2swhs" (OuterVolumeSpecName: "kube-api-access-2swhs") pod "511dcbe5-2703-4084-a081-a87ee369a40a" (UID: "511dcbe5-2703-4084-a081-a87ee369a40a"). InnerVolumeSpecName "kube-api-access-2swhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:47:06.361404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.361386 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae20ece-653f-474b-ba28-202f89e94b2a-kube-api-access-fztwb" (OuterVolumeSpecName: "kube-api-access-fztwb") pod "7ae20ece-653f-474b-ba28-202f89e94b2a" (UID: "7ae20ece-653f-474b-ba28-202f89e94b2a"). InnerVolumeSpecName "kube-api-access-fztwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:47:06.361524 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.361390 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5db11ac-6f16-4272-9917-404552ca181d-kube-api-access-wdbd8" (OuterVolumeSpecName: "kube-api-access-wdbd8") pod "f5db11ac-6f16-4272-9917-404552ca181d" (UID: "f5db11ac-6f16-4272-9917-404552ca181d"). InnerVolumeSpecName "kube-api-access-wdbd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:47:06.361597 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.361582 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ae20ece-653f-474b-ba28-202f89e94b2a" (UID: "7ae20ece-653f-474b-ba28-202f89e94b2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:47:06.460667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460595 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztwb\" (UniqueName: \"kubernetes.io/projected/7ae20ece-653f-474b-ba28-202f89e94b2a-kube-api-access-fztwb\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.460667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460625 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae20ece-653f-474b-ba28-202f89e94b2a-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.460667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460638 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae20ece-653f-474b-ba28-202f89e94b2a-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.460667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460650 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swhs\" (UniqueName: \"kubernetes.io/projected/511dcbe5-2703-4084-a081-a87ee369a40a-kube-api-access-2swhs\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.460667 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460662 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/511dcbe5-2703-4084-a081-a87ee369a40a-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.460903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460675 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5db11ac-6f16-4272-9917-404552ca181d-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.460903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.460686 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbd8\" (UniqueName: \"kubernetes.io/projected/f5db11ac-6f16-4272-9917-404552ca181d-kube-api-access-wdbd8\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:47:06.677783 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.677749 2565 generic.go:356] "Generic (PLEG): container finished" podID="511dcbe5-2703-4084-a081-a87ee369a40a" containerID="fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa" exitCode=0 Jul 29 07:47:06.677939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.677801 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" Jul 29 07:47:06.677939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.677843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" event={"ID":"511dcbe5-2703-4084-a081-a87ee369a40a","Type":"ContainerDied","Data":"fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa"} Jul 29 07:47:06.677939 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.677921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c" event={"ID":"511dcbe5-2703-4084-a081-a87ee369a40a","Type":"ContainerDied","Data":"d81b19683151613ace66178f731328adebda5165a494b606277aef650dd06fb5"} Jul 29 07:47:06.678127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.677979 2565 scope.go:117] "RemoveContainer" containerID="fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa" Jul 29 07:47:06.679240 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.679214 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5db11ac-6f16-4272-9917-404552ca181d" containerID="cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf" exitCode=0 Jul 29 07:47:06.679346 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.679272 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" Jul 29 07:47:06.679346 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.679284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" event={"ID":"f5db11ac-6f16-4272-9917-404552ca181d","Type":"ContainerDied","Data":"cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf"} Jul 29 07:47:06.679346 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.679335 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc" event={"ID":"f5db11ac-6f16-4272-9917-404552ca181d","Type":"ContainerDied","Data":"812e2cd00b5ac007a8d9030bc9ff8163d04cdd5e606c8d1f7c94b299fe8e658a"} Jul 29 07:47:06.680625 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.680606 2565 generic.go:356] "Generic (PLEG): container finished" podID="7ae20ece-653f-474b-ba28-202f89e94b2a" containerID="61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda" exitCode=0 Jul 29 07:47:06.680720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.680635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" event={"ID":"7ae20ece-653f-474b-ba28-202f89e94b2a","Type":"ContainerDied","Data":"61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda"} Jul 29 07:47:06.680720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.680672 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" Jul 29 07:47:06.680720 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.680697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l" event={"ID":"7ae20ece-653f-474b-ba28-202f89e94b2a","Type":"ContainerDied","Data":"103cbddf89459864c84a17be09786c62000abee81a5eb8b2811b239a6842cb1d"} Jul 29 07:47:06.692228 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.692204 2565 scope.go:117] "RemoveContainer" containerID="fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa" Jul 29 07:47:06.692795 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:47:06.692770 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa\": container with ID starting with fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa not found: ID does not exist" containerID="fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa" Jul 29 07:47:06.692912 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.692802 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa"} err="failed to get container status \"fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa\": rpc error: code = NotFound desc = could not find container \"fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa\": container with ID starting with fba1447cf92eafed711edf708bb248423a327c9749e05c3f6cc44561830e3dfa not found: ID does not exist" Jul 29 07:47:06.692912 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.692819 2565 scope.go:117] "RemoveContainer" containerID="cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf" Jul 29 07:47:06.701350 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.701334 2565 scope.go:117] "RemoveContainer" containerID="cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf" Jul 29 07:47:06.701618 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:47:06.701598 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf\": container with ID starting with cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf not found: ID does not exist" containerID="cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf" Jul 29 07:47:06.701704 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.701623 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf"} err="failed to get container status \"cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf\": rpc error: code = NotFound desc = could not find container \"cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf\": container with ID starting with cdfb6994414062eb900a6c56c4b1aa988373ce7b95d62b42d3d173c9c1b638cf not found: ID does not exist" Jul 29 07:47:06.701704 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.701638 2565 scope.go:117] "RemoveContainer" containerID="61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda" Jul 29 07:47:06.708906 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.708887 2565 scope.go:117] "RemoveContainer" containerID="61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda" Jul 29 07:47:06.709147 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:47:06.709130 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda\": container with ID starting with 61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda not found: ID does not exist" containerID="61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda" Jul 29 07:47:06.709192 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.709152 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda"} err="failed to get container status \"61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda\": rpc error: code = NotFound desc = could not find container \"61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda\": container with ID starting with 61031e5f11085d350f8029f9905468d21ba4a8c3c55186d03c6083b444075cda not found: ID does not exist" Jul 29 07:47:06.727276 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.727229 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c"] Jul 29 07:47:06.733704 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.733686 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2c7c574-65ffffdf86-6qd8c"] Jul 29 07:47:06.748841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.748817 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc"] Jul 29 07:47:06.756409 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.756386 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2c7c574-8597749f48-lb5gc"] Jul 29 07:47:06.775802 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.775771 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l"] Jul 29 07:47:06.778778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:06.778759 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2c7c574-6888694b7-cm68l"] Jul 29 07:47:07.018918 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:07.018860 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511dcbe5-2703-4084-a081-a87ee369a40a" path="/var/lib/kubelet/pods/511dcbe5-2703-4084-a081-a87ee369a40a/volumes" Jul 29 07:47:07.019326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:07.019309 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae20ece-653f-474b-ba28-202f89e94b2a" path="/var/lib/kubelet/pods/7ae20ece-653f-474b-ba28-202f89e94b2a/volumes" Jul 29 07:47:07.019696 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:07.019682 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5db11ac-6f16-4272-9917-404552ca181d" path="/var/lib/kubelet/pods/f5db11ac-6f16-4272-9917-404552ca181d/volumes" Jul 29 07:47:43.564650 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.564620 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5"] Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.564980 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511dcbe5-2703-4084-a081-a87ee369a40a" containerName="payload-processing" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.564993 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="511dcbe5-2703-4084-a081-a87ee369a40a" containerName="payload-processing" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565000 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae20ece-653f-474b-ba28-202f89e94b2a" containerName="maas-api" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565008 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae20ece-653f-474b-ba28-202f89e94b2a" containerName="maas-api" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565019 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5db11ac-6f16-4272-9917-404552ca181d" containerName="payload-pre-processing" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565025 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5db11ac-6f16-4272-9917-404552ca181d" containerName="payload-pre-processing" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565097 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="511dcbe5-2703-4084-a081-a87ee369a40a" containerName="payload-processing" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565108 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5db11ac-6f16-4272-9917-404552ca181d" containerName="payload-pre-processing" Jul 29 07:47:43.565142 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.565115 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae20ece-653f-474b-ba28-202f89e94b2a" containerName="maas-api" Jul 29 07:47:43.568089 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.568071 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.572208 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.572180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ca21af-1a5d-478a-80df-249533d6e991-openshift-service-ca\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.572318 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.572232 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svss4\" (UniqueName: \"kubernetes.io/projected/56ca21af-1a5d-478a-80df-249533d6e991-kube-api-access-svss4\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.572318 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.572256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ca21af-1a5d-478a-80df-249533d6e991-maas-api-tls\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.579027 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.579007 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:47:43.579160 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.579065 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:47:43.580454 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.580432 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-412a3f1a-serving-cert\"" Jul 29 07:47:43.587572 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.587555 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:47:43.673534 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.673504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ca21af-1a5d-478a-80df-249533d6e991-maas-api-tls\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.673723 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.673582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ca21af-1a5d-478a-80df-249533d6e991-openshift-service-ca\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.673723 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.673615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svss4\" (UniqueName: \"kubernetes.io/projected/56ca21af-1a5d-478a-80df-249533d6e991-kube-api-access-svss4\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.674275 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.674252 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ca21af-1a5d-478a-80df-249533d6e991-openshift-service-ca\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.675996 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.675979 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ca21af-1a5d-478a-80df-249533d6e991-maas-api-tls\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.690866 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.690838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svss4\" (UniqueName: \"kubernetes.io/projected/56ca21af-1a5d-478a-80df-249533d6e991-kube-api-access-svss4\") pod \"maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:43.725416 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.725388 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5"] Jul 29 07:47:43.878190 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:43.878109 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:44.087579 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:44.087555 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5"] Jul 29 07:47:44.090033 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:47:44.090008 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ca21af_1a5d_478a_80df_249533d6e991.slice/crio-40201e4b779db6e9ef89d50f6181d5f0f7ec599f11bd6038200ce00745fe0aa7 WatchSource:0}: Error finding container 40201e4b779db6e9ef89d50f6181d5f0f7ec599f11bd6038200ce00745fe0aa7: Status 404 returned error can't find the container with id 40201e4b779db6e9ef89d50f6181d5f0f7ec599f11bd6038200ce00745fe0aa7 Jul 29 07:47:44.876466 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:44.876427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" event={"ID":"56ca21af-1a5d-478a-80df-249533d6e991","Type":"ContainerStarted","Data":"8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf"} Jul 29 07:47:44.876887 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:44.876490 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:44.876887 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:44.876509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" event={"ID":"56ca21af-1a5d-478a-80df-249533d6e991","Type":"ContainerStarted","Data":"40201e4b779db6e9ef89d50f6181d5f0f7ec599f11bd6038200ce00745fe0aa7"} Jul 29 07:47:44.932913 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:44.932863 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" podStartSLOduration=1.485061771 podStartE2EDuration="1.932846433s" totalImagesPullingTime="447.784662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:47:44.932489405 +0000 UTC m=+1946.538667514" watchObservedRunningTime="2026-07-29 07:47:44.932846433 +0000 UTC m=+1946.539024540" Jul 29 07:47:45.558376 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.558338 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg"] Jul 29 07:47:45.561771 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.561755 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.565625 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.565601 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-412a3f1a-dockercfg-l5gc4\"" Jul 29 07:47:45.565771 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.565685 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-412a3f1a\"" Jul 29 07:47:45.589784 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.589764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssg5\" (UniqueName: \"kubernetes.io/projected/654452da-9d77-4bc6-bce5-604d1f5d18c0-kube-api-access-sssg5\") pod \"payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.589904 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.589830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654452da-9d77-4bc6-bce5-604d1f5d18c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.600827 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.600801 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg"] Jul 29 07:47:45.655940 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.655911 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56"] Jul 29 07:47:45.659338 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.659323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.690545 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.690517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654452da-9d77-4bc6-bce5-604d1f5d18c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.690658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.690574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rm2\" (UniqueName: \"kubernetes.io/projected/2bc86563-7e78-4d3a-aa68-a016952098b7-kube-api-access-q6rm2\") pod \"payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.690658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.690594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sssg5\" (UniqueName: \"kubernetes.io/projected/654452da-9d77-4bc6-bce5-604d1f5d18c0-kube-api-access-sssg5\") pod \"payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.690658 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.690649 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc86563-7e78-4d3a-aa68-a016952098b7-plugins-config-volume\") pod \"payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.691358 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.691328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654452da-9d77-4bc6-bce5-604d1f5d18c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.704183 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.704160 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56"] Jul 29 07:47:45.737701 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.737674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssg5\" (UniqueName: \"kubernetes.io/projected/654452da-9d77-4bc6-bce5-604d1f5d18c0-kube-api-access-sssg5\") pod \"payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.791841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.791814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rm2\" (UniqueName: \"kubernetes.io/projected/2bc86563-7e78-4d3a-aa68-a016952098b7-kube-api-access-q6rm2\") pod \"payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.792004 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.791872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc86563-7e78-4d3a-aa68-a016952098b7-plugins-config-volume\") pod \"payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.792558 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.792540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc86563-7e78-4d3a-aa68-a016952098b7-plugins-config-volume\") pod \"payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.809255 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.809201 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rm2\" (UniqueName: \"kubernetes.io/projected/2bc86563-7e78-4d3a-aa68-a016952098b7-kube-api-access-q6rm2\") pod \"payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:45.872070 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.872045 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:45.969590 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:45.969548 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:46.134314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.134245 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56"] Jul 29 07:47:46.141913 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:47:46.141886 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc86563_7e78_4d3a_aa68_a016952098b7.slice/crio-29d6525286356882678704cadeba0ea7148ee736edbed30a263eb57f4ed31f07 WatchSource:0}: Error finding container 29d6525286356882678704cadeba0ea7148ee736edbed30a263eb57f4ed31f07: Status 404 returned error can't find the container with id 29d6525286356882678704cadeba0ea7148ee736edbed30a263eb57f4ed31f07 Jul 29 07:47:46.228293 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:47:46.228263 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654452da_9d77_4bc6_bce5_604d1f5d18c0.slice/crio-7837dd4dac132d466f99db41d82d9e2b0673a3bd9b677c0474c02e15998673c3 WatchSource:0}: Error finding container 7837dd4dac132d466f99db41d82d9e2b0673a3bd9b677c0474c02e15998673c3: Status 404 returned error can't find the container with id 7837dd4dac132d466f99db41d82d9e2b0673a3bd9b677c0474c02e15998673c3 Jul 29 07:47:46.230552 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.230523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg"] Jul 29 07:47:46.888215 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.888178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" event={"ID":"654452da-9d77-4bc6-bce5-604d1f5d18c0","Type":"ContainerStarted","Data":"23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01"} Jul 29 07:47:46.888400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.888240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" event={"ID":"654452da-9d77-4bc6-bce5-604d1f5d18c0","Type":"ContainerStarted","Data":"7837dd4dac132d466f99db41d82d9e2b0673a3bd9b677c0474c02e15998673c3"} Jul 29 07:47:46.888400 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.888271 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:46.889778 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.889754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" event={"ID":"2bc86563-7e78-4d3a-aa68-a016952098b7","Type":"ContainerStarted","Data":"571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024"} Jul 29 07:47:46.889895 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.889797 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:47:46.889895 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.889813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" event={"ID":"2bc86563-7e78-4d3a-aa68-a016952098b7","Type":"ContainerStarted","Data":"29d6525286356882678704cadeba0ea7148ee736edbed30a263eb57f4ed31f07"} Jul 29 07:47:46.913876 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.913828 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" podStartSLOduration=1.9138154790000002 podStartE2EDuration="1.913815479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:47:46.909869087 +0000 UTC m=+1948.516047195" watchObservedRunningTime="2026-07-29 07:47:46.913815479 +0000 UTC m=+1948.519993592" Jul 29 07:47:46.951367 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:46.951315 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" podStartSLOduration=1.951296373 podStartE2EDuration="1.951296373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:47:46.944772923 +0000 UTC m=+1948.550951027" watchObservedRunningTime="2026-07-29 07:47:46.951296373 +0000 UTC m=+1948.557474484" Jul 29 07:47:50.887305 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:50.887278 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:47:57.898059 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:57.898031 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:47:57.898497 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:47:57.898098 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:48:45.402728 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.402695 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5"] Jul 29 07:48:45.403203 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.402921 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" podUID="56ca21af-1a5d-478a-80df-249533d6e991" containerName="maas-api" containerID="cri-o://8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf" gracePeriod=30 Jul 29 07:48:45.563640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.563592 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56"] Jul 29 07:48:45.563894 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.563867 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" podUID="2bc86563-7e78-4d3a-aa68-a016952098b7" containerName="payload-processing" containerID="cri-o://571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024" gracePeriod=30 Jul 29 07:48:45.578428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.578000 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg"] Jul 29 07:48:45.578428 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.578228 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" podUID="654452da-9d77-4bc6-bce5-604d1f5d18c0" containerName="payload-pre-processing" containerID="cri-o://23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01" gracePeriod=30 Jul 29 07:48:45.709734 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.709688 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:48:45.800551 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.800515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ca21af-1a5d-478a-80df-249533d6e991-openshift-service-ca\") pod \"56ca21af-1a5d-478a-80df-249533d6e991\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " Jul 29 07:48:45.800747 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.800676 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ca21af-1a5d-478a-80df-249533d6e991-maas-api-tls\") pod \"56ca21af-1a5d-478a-80df-249533d6e991\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " Jul 29 07:48:45.800747 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.800721 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svss4\" (UniqueName: \"kubernetes.io/projected/56ca21af-1a5d-478a-80df-249533d6e991-kube-api-access-svss4\") pod \"56ca21af-1a5d-478a-80df-249533d6e991\" (UID: \"56ca21af-1a5d-478a-80df-249533d6e991\") " Jul 29 07:48:45.800942 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.800897 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ca21af-1a5d-478a-80df-249533d6e991-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56ca21af-1a5d-478a-80df-249533d6e991" (UID: "56ca21af-1a5d-478a-80df-249533d6e991"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:48:45.801088 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.801070 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56ca21af-1a5d-478a-80df-249533d6e991-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:45.803445 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.803402 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ca21af-1a5d-478a-80df-249533d6e991-kube-api-access-svss4" (OuterVolumeSpecName: "kube-api-access-svss4") pod "56ca21af-1a5d-478a-80df-249533d6e991" (UID: "56ca21af-1a5d-478a-80df-249533d6e991"). InnerVolumeSpecName "kube-api-access-svss4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:48:45.803551 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.803455 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ca21af-1a5d-478a-80df-249533d6e991-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56ca21af-1a5d-478a-80df-249533d6e991" (UID: "56ca21af-1a5d-478a-80df-249533d6e991"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:48:45.813840 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.813820 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:48:45.819847 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.819827 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:48:45.901753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.901724 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc86563-7e78-4d3a-aa68-a016952098b7-plugins-config-volume\") pod \"2bc86563-7e78-4d3a-aa68-a016952098b7\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " Jul 29 07:48:45.901923 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.901817 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654452da-9d77-4bc6-bce5-604d1f5d18c0-plugins-config-volume\") pod \"654452da-9d77-4bc6-bce5-604d1f5d18c0\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " Jul 29 07:48:45.901923 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.901896 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6rm2\" (UniqueName: \"kubernetes.io/projected/2bc86563-7e78-4d3a-aa68-a016952098b7-kube-api-access-q6rm2\") pod \"2bc86563-7e78-4d3a-aa68-a016952098b7\" (UID: \"2bc86563-7e78-4d3a-aa68-a016952098b7\") " Jul 29 07:48:45.902055 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.901925 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sssg5\" (UniqueName: \"kubernetes.io/projected/654452da-9d77-4bc6-bce5-604d1f5d18c0-kube-api-access-sssg5\") pod \"654452da-9d77-4bc6-bce5-604d1f5d18c0\" (UID: \"654452da-9d77-4bc6-bce5-604d1f5d18c0\") " Jul 29 07:48:45.902515 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.902253 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56ca21af-1a5d-478a-80df-249533d6e991-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:45.902515 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.902278 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svss4\" (UniqueName: \"kubernetes.io/projected/56ca21af-1a5d-478a-80df-249533d6e991-kube-api-access-svss4\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:45.902515 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.902373 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc86563-7e78-4d3a-aa68-a016952098b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bc86563-7e78-4d3a-aa68-a016952098b7" (UID: "2bc86563-7e78-4d3a-aa68-a016952098b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:48:45.902744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.902549 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654452da-9d77-4bc6-bce5-604d1f5d18c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "654452da-9d77-4bc6-bce5-604d1f5d18c0" (UID: "654452da-9d77-4bc6-bce5-604d1f5d18c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:48:45.904872 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.904835 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc86563-7e78-4d3a-aa68-a016952098b7-kube-api-access-q6rm2" (OuterVolumeSpecName: "kube-api-access-q6rm2") pod "2bc86563-7e78-4d3a-aa68-a016952098b7" (UID: "2bc86563-7e78-4d3a-aa68-a016952098b7"). InnerVolumeSpecName "kube-api-access-q6rm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:48:45.905140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:45.905088 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654452da-9d77-4bc6-bce5-604d1f5d18c0-kube-api-access-sssg5" (OuterVolumeSpecName: "kube-api-access-sssg5") pod "654452da-9d77-4bc6-bce5-604d1f5d18c0" (UID: "654452da-9d77-4bc6-bce5-604d1f5d18c0"). InnerVolumeSpecName "kube-api-access-sssg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:48:46.003486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.003417 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654452da-9d77-4bc6-bce5-604d1f5d18c0-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:46.003486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.003444 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rm2\" (UniqueName: \"kubernetes.io/projected/2bc86563-7e78-4d3a-aa68-a016952098b7-kube-api-access-q6rm2\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:46.003486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.003454 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sssg5\" (UniqueName: \"kubernetes.io/projected/654452da-9d77-4bc6-bce5-604d1f5d18c0-kube-api-access-sssg5\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:46.003486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.003462 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc86563-7e78-4d3a-aa68-a016952098b7-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:48:46.194748 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.194707 2565 generic.go:356] "Generic (PLEG): container finished" podID="2bc86563-7e78-4d3a-aa68-a016952098b7" containerID="571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024" exitCode=0 Jul 29 07:48:46.195019 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.194752 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" Jul 29 07:48:46.195019 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.194800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" event={"ID":"2bc86563-7e78-4d3a-aa68-a016952098b7","Type":"ContainerDied","Data":"571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024"} Jul 29 07:48:46.195019 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.194851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56" event={"ID":"2bc86563-7e78-4d3a-aa68-a016952098b7","Type":"ContainerDied","Data":"29d6525286356882678704cadeba0ea7148ee736edbed30a263eb57f4ed31f07"} Jul 29 07:48:46.195019 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.194875 2565 scope.go:117] "RemoveContainer" containerID="571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024" Jul 29 07:48:46.196218 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.196191 2565 generic.go:356] "Generic (PLEG): container finished" podID="56ca21af-1a5d-478a-80df-249533d6e991" containerID="8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf" exitCode=0 Jul 29 07:48:46.196348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.196220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" event={"ID":"56ca21af-1a5d-478a-80df-249533d6e991","Type":"ContainerDied","Data":"8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf"} Jul 29 07:48:46.196348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.196243 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" Jul 29 07:48:46.196348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.196266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5" event={"ID":"56ca21af-1a5d-478a-80df-249533d6e991","Type":"ContainerDied","Data":"40201e4b779db6e9ef89d50f6181d5f0f7ec599f11bd6038200ce00745fe0aa7"} Jul 29 07:48:46.197827 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.197801 2565 generic.go:356] "Generic (PLEG): container finished" podID="654452da-9d77-4bc6-bce5-604d1f5d18c0" containerID="23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01" exitCode=0 Jul 29 07:48:46.197941 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.197835 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" event={"ID":"654452da-9d77-4bc6-bce5-604d1f5d18c0","Type":"ContainerDied","Data":"23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01"} Jul 29 07:48:46.197941 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.197847 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" Jul 29 07:48:46.197941 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.197874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg" event={"ID":"654452da-9d77-4bc6-bce5-604d1f5d18c0","Type":"ContainerDied","Data":"7837dd4dac132d466f99db41d82d9e2b0673a3bd9b677c0474c02e15998673c3"} Jul 29 07:48:46.205486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.205463 2565 scope.go:117] "RemoveContainer" containerID="571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024" Jul 29 07:48:46.205755 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:48:46.205732 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024\": container with ID starting with 571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024 not found: ID does not exist" containerID="571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024" Jul 29 07:48:46.205835 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.205765 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024"} err="failed to get container status \"571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024\": rpc error: code = NotFound desc = could not find container \"571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024\": container with ID starting with 571ffc9a9adbbe4b15e4542c3d5c5079ad43caac3174f31f6cd511d2dbad4024 not found: ID does not exist" Jul 29 07:48:46.205835 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.205781 2565 scope.go:117] "RemoveContainer" containerID="8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf" Jul 29 07:48:46.213753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.213735 2565 scope.go:117] "RemoveContainer" containerID="8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf" Jul 29 07:48:46.214075 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:48:46.214054 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf\": container with ID starting with 8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf not found: ID does not exist" containerID="8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf" Jul 29 07:48:46.214145 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.214080 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf"} err="failed to get container status \"8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf\": rpc error: code = NotFound desc = could not find container \"8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf\": container with ID starting with 8e9044efc06d6f0b206b6633deb26d6ab0e11bab26e1e9213c8398054d1d1aaf not found: ID does not exist" Jul 29 07:48:46.214145 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.214096 2565 scope.go:117] "RemoveContainer" containerID="23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01" Jul 29 07:48:46.221366 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.221350 2565 scope.go:117] "RemoveContainer" containerID="23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01" Jul 29 07:48:46.221604 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:48:46.221586 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01\": container with ID starting with 23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01 not found: ID does not exist" containerID="23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01" Jul 29 07:48:46.221659 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.221610 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01"} err="failed to get container status \"23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01\": rpc error: code = NotFound desc = could not find container \"23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01\": container with ID starting with 23ed442734937f145ce0bf0c4153674507af2caf2ecf770f0c7a2ce2e3e4af01 not found: ID does not exist" Jul 29 07:48:46.270565 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.264280 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5"] Jul 29 07:48:46.275015 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.274993 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-412a3f1a-746bcb67d4-2fmm5"] Jul 29 07:48:46.328225 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.328195 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg"] Jul 29 07:48:46.342638 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.342610 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-412a3f1a-6dbd8574c6-ddvfg"] Jul 29 07:48:46.371999 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.371967 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56"] Jul 29 07:48:46.383424 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:46.383394 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-412a3f1a-5b6467d84b-p4v56"] Jul 29 07:48:47.022156 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:47.022092 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc86563-7e78-4d3a-aa68-a016952098b7" path="/var/lib/kubelet/pods/2bc86563-7e78-4d3a-aa68-a016952098b7/volumes" Jul 29 07:48:47.023343 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:47.023318 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ca21af-1a5d-478a-80df-249533d6e991" path="/var/lib/kubelet/pods/56ca21af-1a5d-478a-80df-249533d6e991/volumes" Jul 29 07:48:47.024043 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:48:47.024024 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654452da-9d77-4bc6-bce5-604d1f5d18c0" path="/var/lib/kubelet/pods/654452da-9d77-4bc6-bce5-604d1f5d18c0/volumes" Jul 29 07:50:48.168121 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168087 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn"] Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168419 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc86563-7e78-4d3a-aa68-a016952098b7" containerName="payload-processing" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168429 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc86563-7e78-4d3a-aa68-a016952098b7" containerName="payload-processing" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168445 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ca21af-1a5d-478a-80df-249533d6e991" containerName="maas-api" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168450 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ca21af-1a5d-478a-80df-249533d6e991" containerName="maas-api" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168468 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654452da-9d77-4bc6-bce5-604d1f5d18c0" containerName="payload-pre-processing" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168473 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="654452da-9d77-4bc6-bce5-604d1f5d18c0" containerName="payload-pre-processing" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168525 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ca21af-1a5d-478a-80df-249533d6e991" containerName="maas-api" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168532 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc86563-7e78-4d3a-aa68-a016952098b7" containerName="payload-processing" Jul 29 07:50:48.168560 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.168537 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="654452da-9d77-4bc6-bce5-604d1f5d18c0" containerName="payload-pre-processing" Jul 29 07:50:48.171404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.171388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.188085 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.188062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5v8\" (UniqueName: \"kubernetes.io/projected/ef6b2f14-944c-431b-8994-231d223e3eed-kube-api-access-xb5v8\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.188209 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.188113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.188209 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.188147 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b2f14-944c-431b-8994-231d223e3eed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.190477 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.190458 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7d6491-serving-cert\"" Jul 29 07:50:48.192990 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.192966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:50:48.193093 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.193064 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:50:48.193193 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.193178 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:50:48.256080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.256059 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn"] Jul 29 07:50:48.288718 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.288694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5v8\" (UniqueName: \"kubernetes.io/projected/ef6b2f14-944c-431b-8994-231d223e3eed-kube-api-access-xb5v8\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.288831 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.288735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.288831 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.288764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b2f14-944c-431b-8994-231d223e3eed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.288978 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:50:48.288907 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-serving-cert: secret "maas-api-e2e-inf-a-7d6491-serving-cert" not found Jul 29 07:50:48.289052 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:50:48.289023 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls podName:ef6b2f14-944c-431b-8994-231d223e3eed nodeName:}" failed. No retries permitted until 2026-07-29 07:50:48.788999445 +0000 UTC m=+2130.395177541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls") pod "maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" (UID: "ef6b2f14-944c-431b-8994-231d223e3eed") : secret "maas-api-e2e-inf-a-7d6491-serving-cert" not found Jul 29 07:50:48.289379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.289361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b2f14-944c-431b-8994-231d223e3eed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.422887 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.422819 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5v8\" (UniqueName: \"kubernetes.io/projected/ef6b2f14-944c-431b-8994-231d223e3eed-kube-api-access-xb5v8\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.793453 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.793370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:48.795677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:48.795657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls\") pod \"maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:49.082510 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:49.082481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:49.318801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:49.318773 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn"] Jul 29 07:50:49.320396 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:50:49.320368 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6b2f14_944c_431b_8994_231d223e3eed.slice/crio-8f6c5edf2cbcc77cc0a3717413dcb65ceed5e86b6e18a5c06298a5c1f3681e00 WatchSource:0}: Error finding container 8f6c5edf2cbcc77cc0a3717413dcb65ceed5e86b6e18a5c06298a5c1f3681e00: Status 404 returned error can't find the container with id 8f6c5edf2cbcc77cc0a3717413dcb65ceed5e86b6e18a5c06298a5c1f3681e00 Jul 29 07:50:49.321733 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:49.321714 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:50:49.832946 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:49.832905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" event={"ID":"ef6b2f14-944c-431b-8994-231d223e3eed","Type":"ContainerStarted","Data":"8f6c5edf2cbcc77cc0a3717413dcb65ceed5e86b6e18a5c06298a5c1f3681e00"} Jul 29 07:50:50.302744 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.302643 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt"] Jul 29 07:50:50.306231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.306208 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.310876 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.310850 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7d6491\"" Jul 29 07:50:50.311014 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.310947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7d6491-dockercfg-sw8ms\"" Jul 29 07:50:50.339167 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.339145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt"] Jul 29 07:50:50.408047 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.408014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbtk\" (UniqueName: \"kubernetes.io/projected/c056237c-91f5-408d-9ed8-a22ca993ca0d-kube-api-access-cpbtk\") pod \"payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.408047 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.408046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c056237c-91f5-408d-9ed8-a22ca993ca0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.509189 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.509155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbtk\" (UniqueName: \"kubernetes.io/projected/c056237c-91f5-408d-9ed8-a22ca993ca0d-kube-api-access-cpbtk\") pod \"payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.509189 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.509194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c056237c-91f5-408d-9ed8-a22ca993ca0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.509925 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.509906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c056237c-91f5-408d-9ed8-a22ca993ca0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.524024 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.523994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbtk\" (UniqueName: \"kubernetes.io/projected/c056237c-91f5-408d-9ed8-a22ca993ca0d-kube-api-access-cpbtk\") pod \"payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.559445 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.559419 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d"] Jul 29 07:50:50.563033 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.563016 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.592389 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.592363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d"] Jul 29 07:50:50.610296 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.610273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86v7\" (UniqueName: \"kubernetes.io/projected/f07663c8-53e6-4de4-828f-25b8a5160265-kube-api-access-m86v7\") pod \"payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.610413 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.610339 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07663c8-53e6-4de4-828f-25b8a5160265-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.616135 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.616106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.711628 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.711588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86v7\" (UniqueName: \"kubernetes.io/projected/f07663c8-53e6-4de4-828f-25b8a5160265-kube-api-access-m86v7\") pod \"payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.711762 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.711747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07663c8-53e6-4de4-828f-25b8a5160265-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.712692 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.712669 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07663c8-53e6-4de4-828f-25b8a5160265-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.729482 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.729452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86v7\" (UniqueName: \"kubernetes.io/projected/f07663c8-53e6-4de4-828f-25b8a5160265-kube-api-access-m86v7\") pod \"payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.767127 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.767106 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt"] Jul 29 07:50:50.768062 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:50:50.768032 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc056237c_91f5_408d_9ed8_a22ca993ca0d.slice/crio-0c54925ba1dba7d7bfa221ba9a69a9e2aa9b20ecedc68baf9432ec310fc0de78 WatchSource:0}: Error finding container 0c54925ba1dba7d7bfa221ba9a69a9e2aa9b20ecedc68baf9432ec310fc0de78: Status 404 returned error can't find the container with id 0c54925ba1dba7d7bfa221ba9a69a9e2aa9b20ecedc68baf9432ec310fc0de78 Jul 29 07:50:50.839535 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.839505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" event={"ID":"c056237c-91f5-408d-9ed8-a22ca993ca0d","Type":"ContainerStarted","Data":"b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b"} Jul 29 07:50:50.839657 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.839565 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:50:50.839657 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.839589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" event={"ID":"c056237c-91f5-408d-9ed8-a22ca993ca0d","Type":"ContainerStarted","Data":"0c54925ba1dba7d7bfa221ba9a69a9e2aa9b20ecedc68baf9432ec310fc0de78"} Jul 29 07:50:50.840948 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.840927 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" event={"ID":"ef6b2f14-944c-431b-8994-231d223e3eed","Type":"ContainerStarted","Data":"3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2"} Jul 29 07:50:50.841049 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.840992 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:50:50.873360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.873341 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:50.903612 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.903560 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" podStartSLOduration=0.903546653 podStartE2EDuration="903.546653ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:50:50.901553363 +0000 UTC m=+2132.507731472" watchObservedRunningTime="2026-07-29 07:50:50.903546653 +0000 UTC m=+2132.509724780" Jul 29 07:50:50.982945 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:50.982897 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" podStartSLOduration=2.407793759 podStartE2EDuration="2.982880307s" totalImagesPullingTime="575.086548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:50:50.952800339 +0000 UTC m=+2132.558978452" watchObservedRunningTime="2026-07-29 07:50:50.982880307 +0000 UTC m=+2132.589058415" Jul 29 07:50:51.220349 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:51.220319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d"] Jul 29 07:50:51.220893 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:50:51.220865 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07663c8_53e6_4de4_828f_25b8a5160265.slice/crio-108e3ecca18a0d406aa4a58541bf0bb92eb4596c80763ef0a12b42d8acee87b6 WatchSource:0}: Error finding container 108e3ecca18a0d406aa4a58541bf0bb92eb4596c80763ef0a12b42d8acee87b6: Status 404 returned error can't find the container with id 108e3ecca18a0d406aa4a58541bf0bb92eb4596c80763ef0a12b42d8acee87b6 Jul 29 07:50:51.848714 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:51.848667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" event={"ID":"f07663c8-53e6-4de4-828f-25b8a5160265","Type":"ContainerStarted","Data":"a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc"} Jul 29 07:50:51.849215 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:51.848745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" event={"ID":"f07663c8-53e6-4de4-828f-25b8a5160265","Type":"ContainerStarted","Data":"108e3ecca18a0d406aa4a58541bf0bb92eb4596c80763ef0a12b42d8acee87b6"} Jul 29 07:50:51.849215 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:51.848938 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:50:51.879759 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:51.879714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" podStartSLOduration=1.87970007 podStartE2EDuration="1.87970007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:50:51.877764056 +0000 UTC m=+2133.483942177" watchObservedRunningTime="2026-07-29 07:50:51.87970007 +0000 UTC m=+2133.485878177" Jul 29 07:50:56.854015 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:50:56.853974 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:51:01.851017 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:01.850977 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" podUID="c056237c-91f5-408d-9ed8-a22ca993ca0d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.60:9004: connect: connection refused" Jul 29 07:51:02.857735 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:02.857707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:51:11.849827 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:11.849749 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:51:17.210933 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.210903 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7"] Jul 29 07:51:17.214404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.214388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.218164 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.218142 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d6451b-serving-cert\"" Jul 29 07:51:17.243465 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.243440 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7"] Jul 29 07:51:17.343011 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.342987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b64fd9-e567-4a7f-8729-97320c956a65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.343135 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.343032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b64fd9-e567-4a7f-8729-97320c956a65-maas-api-tls\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.343179 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.343145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbv2t\" (UniqueName: \"kubernetes.io/projected/87b64fd9-e567-4a7f-8729-97320c956a65-kube-api-access-wbv2t\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.444128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.444098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b64fd9-e567-4a7f-8729-97320c956a65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.444290 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.444153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b64fd9-e567-4a7f-8729-97320c956a65-maas-api-tls\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.444290 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.444228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbv2t\" (UniqueName: \"kubernetes.io/projected/87b64fd9-e567-4a7f-8729-97320c956a65-kube-api-access-wbv2t\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.444736 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.444716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b64fd9-e567-4a7f-8729-97320c956a65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.446679 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.446657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b64fd9-e567-4a7f-8729-97320c956a65-maas-api-tls\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.481249 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.481182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbv2t\" (UniqueName: \"kubernetes.io/projected/87b64fd9-e567-4a7f-8729-97320c956a65-kube-api-access-wbv2t\") pod \"maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.524703 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.524685 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:17.663568 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.663545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7"] Jul 29 07:51:17.664519 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:51:17.664491 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b64fd9_e567_4a7f_8729_97320c956a65.slice/crio-5f53c63479cce47ab22c1897bc957a91a352daa8161cf2e60bc12d13c468e38c WatchSource:0}: Error finding container 5f53c63479cce47ab22c1897bc957a91a352daa8161cf2e60bc12d13c468e38c: Status 404 returned error can't find the container with id 5f53c63479cce47ab22c1897bc957a91a352daa8161cf2e60bc12d13c468e38c Jul 29 07:51:17.990650 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:17.990616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" event={"ID":"87b64fd9-e567-4a7f-8729-97320c956a65","Type":"ContainerStarted","Data":"5f53c63479cce47ab22c1897bc957a91a352daa8161cf2e60bc12d13c468e38c"} Jul 29 07:51:18.257197 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.257104 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8"] Jul 29 07:51:18.261821 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.261790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.266307 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.266281 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d6451b\"" Jul 29 07:51:18.266452 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.266348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d6451b-dockercfg-jpq2v\"" Jul 29 07:51:18.274480 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.274445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8"] Jul 29 07:51:18.292501 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.292476 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49"] Jul 29 07:51:18.296239 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.296216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.314079 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.314055 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49"] Jul 29 07:51:18.353929 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.353901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4nbp\" (UniqueName: \"kubernetes.io/projected/57a2a482-79f6-4eda-a025-f6b42562b1b7-kube-api-access-j4nbp\") pod \"payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.354079 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.353976 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2a482-79f6-4eda-a025-f6b42562b1b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.455108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.455077 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rq5\" (UniqueName: \"kubernetes.io/projected/2b9bb568-6988-4330-a421-ea0fa9d27c04-kube-api-access-s5rq5\") pod \"payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.455369 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.455349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4nbp\" (UniqueName: \"kubernetes.io/projected/57a2a482-79f6-4eda-a025-f6b42562b1b7-kube-api-access-j4nbp\") pod \"payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.455529 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.455514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2a482-79f6-4eda-a025-f6b42562b1b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.455672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.455656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9bb568-6988-4330-a421-ea0fa9d27c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.457074 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.457049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2a482-79f6-4eda-a025-f6b42562b1b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.465302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.465275 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4nbp\" (UniqueName: \"kubernetes.io/projected/57a2a482-79f6-4eda-a025-f6b42562b1b7-kube-api-access-j4nbp\") pod \"payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.556200 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.556176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rq5\" (UniqueName: \"kubernetes.io/projected/2b9bb568-6988-4330-a421-ea0fa9d27c04-kube-api-access-s5rq5\") pod \"payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.556330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.556250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9bb568-6988-4330-a421-ea0fa9d27c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.556873 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.556854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9bb568-6988-4330-a421-ea0fa9d27c04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.564725 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.564697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rq5\" (UniqueName: \"kubernetes.io/projected/2b9bb568-6988-4330-a421-ea0fa9d27c04-kube-api-access-s5rq5\") pod \"payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.579122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.579096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.609572 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.609546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:18.723076 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.723041 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8"] Jul 29 07:51:18.723917 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:51:18.723886 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a2a482_79f6_4eda_a025_f6b42562b1b7.slice/crio-8f1560a4860b32c411a7c0cb46b7ca3f69379ef8b2164ee50b06b5f84c0c4c2b WatchSource:0}: Error finding container 8f1560a4860b32c411a7c0cb46b7ca3f69379ef8b2164ee50b06b5f84c0c4c2b: Status 404 returned error can't find the container with id 8f1560a4860b32c411a7c0cb46b7ca3f69379ef8b2164ee50b06b5f84c0c4c2b Jul 29 07:51:18.759876 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.759849 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49"] Jul 29 07:51:18.760558 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:51:18.760537 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9bb568_6988_4330_a421_ea0fa9d27c04.slice/crio-21bec44e4bdfbc8dd0cc4aea7674df3fbeb2e17b56cd12ac39942e665a2f2990 WatchSource:0}: Error finding container 21bec44e4bdfbc8dd0cc4aea7674df3fbeb2e17b56cd12ac39942e665a2f2990: Status 404 returned error can't find the container with id 21bec44e4bdfbc8dd0cc4aea7674df3fbeb2e17b56cd12ac39942e665a2f2990 Jul 29 07:51:18.998581 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.998540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" event={"ID":"57a2a482-79f6-4eda-a025-f6b42562b1b7","Type":"ContainerStarted","Data":"d6ffebb59cd752a640d4f1cadc5008600f28c1264f0438708ff819197aa89125"} Jul 29 07:51:18.998763 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.998705 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:18.998763 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.998737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" event={"ID":"57a2a482-79f6-4eda-a025-f6b42562b1b7","Type":"ContainerStarted","Data":"8f1560a4860b32c411a7c0cb46b7ca3f69379ef8b2164ee50b06b5f84c0c4c2b"} Jul 29 07:51:19.000006 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:18.999975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" event={"ID":"87b64fd9-e567-4a7f-8729-97320c956a65","Type":"ContainerStarted","Data":"f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351"} Jul 29 07:51:19.000122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.000086 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:19.001424 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.001405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" event={"ID":"2b9bb568-6988-4330-a421-ea0fa9d27c04","Type":"ContainerStarted","Data":"25b374661bc2301b278933824d8a20632a0a81c68ea225810bb6fbd68513ad9f"} Jul 29 07:51:19.001502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.001436 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" event={"ID":"2b9bb568-6988-4330-a421-ea0fa9d27c04","Type":"ContainerStarted","Data":"21bec44e4bdfbc8dd0cc4aea7674df3fbeb2e17b56cd12ac39942e665a2f2990"} Jul 29 07:51:19.001502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.001455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:51:19.020220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.020172 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" podStartSLOduration=1.020157311 podStartE2EDuration="1.020157311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:51:19.019658504 +0000 UTC m=+2160.625836601" watchObservedRunningTime="2026-07-29 07:51:19.020157311 +0000 UTC m=+2160.626335421" Jul 29 07:51:19.049841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.049797 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" podStartSLOduration=1.687501994 podStartE2EDuration="2.049784378s" totalImagesPullingTime="362.282384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:51:19.046979344 +0000 UTC m=+2160.653157449" watchObservedRunningTime="2026-07-29 07:51:19.049784378 +0000 UTC m=+2160.655962486" Jul 29 07:51:19.065207 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:19.065126 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" podStartSLOduration=1.065112873 podStartE2EDuration="1.065112873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:51:19.063894731 +0000 UTC m=+2160.670072840" watchObservedRunningTime="2026-07-29 07:51:19.065112873 +0000 UTC m=+2160.671290981" Jul 29 07:51:25.013072 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:25.013039 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:51:30.009737 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:30.009700 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:51:30.010210 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:51:30.009771 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:53:34.976693 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:34.976656 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn"] Jul 29 07:53:34.977200 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:34.976901 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" podUID="ef6b2f14-944c-431b-8994-231d223e3eed" containerName="maas-api" containerID="cri-o://3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2" gracePeriod=30 Jul 29 07:53:35.103000 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.102523 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt"] Jul 29 07:53:35.103000 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.102785 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" podUID="c056237c-91f5-408d-9ed8-a22ca993ca0d" containerName="payload-pre-processing" containerID="cri-o://b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b" gracePeriod=30 Jul 29 07:53:35.105645 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.105619 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d"] Jul 29 07:53:35.106094 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.106062 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" podUID="f07663c8-53e6-4de4-828f-25b8a5160265" containerName="payload-processing" containerID="cri-o://a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc" gracePeriod=30 Jul 29 07:53:35.172176 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:53:35.172134 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07663c8_53e6_4de4_828f_25b8a5160265.slice/crio-a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc.scope\": RecentStats: unable to find data in memory cache]" Jul 29 07:53:35.243016 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.241188 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:53:35.330007 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.329947 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb5v8\" (UniqueName: \"kubernetes.io/projected/ef6b2f14-944c-431b-8994-231d223e3eed-kube-api-access-xb5v8\") pod \"ef6b2f14-944c-431b-8994-231d223e3eed\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " Jul 29 07:53:35.330161 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.330062 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls\") pod \"ef6b2f14-944c-431b-8994-231d223e3eed\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " Jul 29 07:53:35.330161 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.330118 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b2f14-944c-431b-8994-231d223e3eed-openshift-service-ca\") pod \"ef6b2f14-944c-431b-8994-231d223e3eed\" (UID: \"ef6b2f14-944c-431b-8994-231d223e3eed\") " Jul 29 07:53:35.330816 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.330786 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6b2f14-944c-431b-8994-231d223e3eed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef6b2f14-944c-431b-8994-231d223e3eed" (UID: "ef6b2f14-944c-431b-8994-231d223e3eed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:35.333610 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.333570 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef6b2f14-944c-431b-8994-231d223e3eed" (UID: "ef6b2f14-944c-431b-8994-231d223e3eed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:53:35.333905 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.333877 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6b2f14-944c-431b-8994-231d223e3eed-kube-api-access-xb5v8" (OuterVolumeSpecName: "kube-api-access-xb5v8") pod "ef6b2f14-944c-431b-8994-231d223e3eed" (UID: "ef6b2f14-944c-431b-8994-231d223e3eed"). InnerVolumeSpecName "kube-api-access-xb5v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:35.381150 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.381128 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:53:35.385079 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.385061 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:53:35.430867 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.430826 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07663c8-53e6-4de4-828f-25b8a5160265-plugins-config-volume\") pod \"f07663c8-53e6-4de4-828f-25b8a5160265\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " Jul 29 07:53:35.431061 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.430893 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m86v7\" (UniqueName: \"kubernetes.io/projected/f07663c8-53e6-4de4-828f-25b8a5160265-kube-api-access-m86v7\") pod \"f07663c8-53e6-4de4-828f-25b8a5160265\" (UID: \"f07663c8-53e6-4de4-828f-25b8a5160265\") " Jul 29 07:53:35.431186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.431113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c056237c-91f5-408d-9ed8-a22ca993ca0d-plugins-config-volume\") pod \"c056237c-91f5-408d-9ed8-a22ca993ca0d\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " Jul 29 07:53:35.431186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.431147 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpbtk\" (UniqueName: \"kubernetes.io/projected/c056237c-91f5-408d-9ed8-a22ca993ca0d-kube-api-access-cpbtk\") pod \"c056237c-91f5-408d-9ed8-a22ca993ca0d\" (UID: \"c056237c-91f5-408d-9ed8-a22ca993ca0d\") " Jul 29 07:53:35.431446 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.431424 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b2f14-944c-431b-8994-231d223e3eed-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.431545 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.431449 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5v8\" (UniqueName: \"kubernetes.io/projected/ef6b2f14-944c-431b-8994-231d223e3eed-kube-api-access-xb5v8\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.431545 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.431462 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef6b2f14-944c-431b-8994-231d223e3eed-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.432062 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.432022 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c056237c-91f5-408d-9ed8-a22ca993ca0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c056237c-91f5-408d-9ed8-a22ca993ca0d" (UID: "c056237c-91f5-408d-9ed8-a22ca993ca0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:35.432440 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.432410 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07663c8-53e6-4de4-828f-25b8a5160265-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f07663c8-53e6-4de4-828f-25b8a5160265" (UID: "f07663c8-53e6-4de4-828f-25b8a5160265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:35.434009 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.433984 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07663c8-53e6-4de4-828f-25b8a5160265-kube-api-access-m86v7" (OuterVolumeSpecName: "kube-api-access-m86v7") pod "f07663c8-53e6-4de4-828f-25b8a5160265" (UID: "f07663c8-53e6-4de4-828f-25b8a5160265"). InnerVolumeSpecName "kube-api-access-m86v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:35.434112 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.434033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c056237c-91f5-408d-9ed8-a22ca993ca0d-kube-api-access-cpbtk" (OuterVolumeSpecName: "kube-api-access-cpbtk") pod "c056237c-91f5-408d-9ed8-a22ca993ca0d" (UID: "c056237c-91f5-408d-9ed8-a22ca993ca0d"). InnerVolumeSpecName "kube-api-access-cpbtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:35.532320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.532251 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c056237c-91f5-408d-9ed8-a22ca993ca0d-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.532320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.532275 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbtk\" (UniqueName: \"kubernetes.io/projected/c056237c-91f5-408d-9ed8-a22ca993ca0d-kube-api-access-cpbtk\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.532320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.532286 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07663c8-53e6-4de4-828f-25b8a5160265-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.532320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.532295 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86v7\" (UniqueName: \"kubernetes.io/projected/f07663c8-53e6-4de4-828f-25b8a5160265-kube-api-access-m86v7\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:35.713388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.713355 2565 generic.go:356] "Generic (PLEG): container finished" podID="f07663c8-53e6-4de4-828f-25b8a5160265" containerID="a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc" exitCode=0 Jul 29 07:53:35.713550 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.713405 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" Jul 29 07:53:35.713550 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.713444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" event={"ID":"f07663c8-53e6-4de4-828f-25b8a5160265","Type":"ContainerDied","Data":"a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc"} Jul 29 07:53:35.713550 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.713503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d" event={"ID":"f07663c8-53e6-4de4-828f-25b8a5160265","Type":"ContainerDied","Data":"108e3ecca18a0d406aa4a58541bf0bb92eb4596c80763ef0a12b42d8acee87b6"} Jul 29 07:53:35.713550 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.713534 2565 scope.go:117] "RemoveContainer" containerID="a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc" Jul 29 07:53:35.714927 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.714909 2565 generic.go:356] "Generic (PLEG): container finished" podID="ef6b2f14-944c-431b-8994-231d223e3eed" containerID="3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2" exitCode=0 Jul 29 07:53:35.715043 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.714973 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" Jul 29 07:53:35.715043 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.715034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" event={"ID":"ef6b2f14-944c-431b-8994-231d223e3eed","Type":"ContainerDied","Data":"3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2"} Jul 29 07:53:35.715164 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.715076 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn" event={"ID":"ef6b2f14-944c-431b-8994-231d223e3eed","Type":"ContainerDied","Data":"8f6c5edf2cbcc77cc0a3717413dcb65ceed5e86b6e18a5c06298a5c1f3681e00"} Jul 29 07:53:35.716531 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.716508 2565 generic.go:356] "Generic (PLEG): container finished" podID="c056237c-91f5-408d-9ed8-a22ca993ca0d" containerID="b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b" exitCode=0 Jul 29 07:53:35.716636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.716544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" event={"ID":"c056237c-91f5-408d-9ed8-a22ca993ca0d","Type":"ContainerDied","Data":"b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b"} Jul 29 07:53:35.716636 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.716565 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" Jul 29 07:53:35.716743 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.716573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt" event={"ID":"c056237c-91f5-408d-9ed8-a22ca993ca0d","Type":"ContainerDied","Data":"0c54925ba1dba7d7bfa221ba9a69a9e2aa9b20ecedc68baf9432ec310fc0de78"} Jul 29 07:53:35.723351 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.723335 2565 scope.go:117] "RemoveContainer" containerID="a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc" Jul 29 07:53:35.723604 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:53:35.723576 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc\": container with ID starting with a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc not found: ID does not exist" containerID="a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc" Jul 29 07:53:35.723668 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.723611 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc"} err="failed to get container status \"a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc\": rpc error: code = NotFound desc = could not find container \"a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc\": container with ID starting with a71b84a4ebe09678d07e9de497900dc35ee9910b46c7cd07c32b43062cd36ebc not found: ID does not exist" Jul 29 07:53:35.723668 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.723626 2565 scope.go:117] "RemoveContainer" containerID="3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2" Jul 29 07:53:35.732000 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.731984 2565 scope.go:117] "RemoveContainer" containerID="3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2" Jul 29 07:53:35.732239 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:53:35.732221 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2\": container with ID starting with 3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2 not found: ID does not exist" containerID="3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2" Jul 29 07:53:35.732304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.732243 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2"} err="failed to get container status \"3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2\": rpc error: code = NotFound desc = could not find container \"3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2\": container with ID starting with 3a8d70469c939953dafac853059abc81d63f433a06d7384b0d87f95588f5e3d2 not found: ID does not exist" Jul 29 07:53:35.732304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.732258 2565 scope.go:117] "RemoveContainer" containerID="b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b" Jul 29 07:53:35.740343 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.740327 2565 scope.go:117] "RemoveContainer" containerID="b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b" Jul 29 07:53:35.740597 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:53:35.740580 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b\": container with ID starting with b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b not found: ID does not exist" containerID="b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b" Jul 29 07:53:35.740645 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.740603 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b"} err="failed to get container status \"b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b\": rpc error: code = NotFound desc = could not find container \"b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b\": container with ID starting with b04a9dcd89647d62ec6fc619386ae4a7b7c46d62e90134a1ec9a59cd43f8439b not found: ID does not exist" Jul 29 07:53:35.746092 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.746071 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d"] Jul 29 07:53:35.750398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.750379 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d6491-7b8db6b676-z7s9d"] Jul 29 07:53:35.763287 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.763267 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn"] Jul 29 07:53:35.770704 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.770684 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7d6491-5bdbb9d49b-kbgsn"] Jul 29 07:53:35.783190 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.783134 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt"] Jul 29 07:53:35.788140 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:35.788121 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d6491-7f5b868d4d-d6vnt"] Jul 29 07:53:37.019429 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:37.019395 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c056237c-91f5-408d-9ed8-a22ca993ca0d" path="/var/lib/kubelet/pods/c056237c-91f5-408d-9ed8-a22ca993ca0d/volumes" Jul 29 07:53:37.019813 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:37.019799 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6b2f14-944c-431b-8994-231d223e3eed" path="/var/lib/kubelet/pods/ef6b2f14-944c-431b-8994-231d223e3eed/volumes" Jul 29 07:53:37.020148 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:37.020136 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f07663c8-53e6-4de4-828f-25b8a5160265" path="/var/lib/kubelet/pods/f07663c8-53e6-4de4-828f-25b8a5160265/volumes" Jul 29 07:53:54.475360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.475328 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7"] Jul 29 07:53:54.475880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.475553 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" podUID="87b64fd9-e567-4a7f-8729-97320c956a65" containerName="maas-api" containerID="cri-o://f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351" gracePeriod=30 Jul 29 07:53:54.602198 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.602159 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8"] Jul 29 07:53:54.602389 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.602361 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" podUID="57a2a482-79f6-4eda-a025-f6b42562b1b7" containerName="payload-processing" containerID="cri-o://d6ffebb59cd752a640d4f1cadc5008600f28c1264f0438708ff819197aa89125" gracePeriod=30 Jul 29 07:53:54.604881 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.604856 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49"] Jul 29 07:53:54.605182 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.605133 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" podUID="2b9bb568-6988-4330-a421-ea0fa9d27c04" containerName="payload-pre-processing" containerID="cri-o://25b374661bc2301b278933824d8a20632a0a81c68ea225810bb6fbd68513ad9f" gracePeriod=30 Jul 29 07:53:54.744187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.744153 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:53:54.826267 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.826213 2565 generic.go:356] "Generic (PLEG): container finished" podID="2b9bb568-6988-4330-a421-ea0fa9d27c04" containerID="25b374661bc2301b278933824d8a20632a0a81c68ea225810bb6fbd68513ad9f" exitCode=0 Jul 29 07:53:54.826513 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.826403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" event={"ID":"2b9bb568-6988-4330-a421-ea0fa9d27c04","Type":"ContainerDied","Data":"25b374661bc2301b278933824d8a20632a0a81c68ea225810bb6fbd68513ad9f"} Jul 29 07:53:54.828422 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.828393 2565 generic.go:356] "Generic (PLEG): container finished" podID="57a2a482-79f6-4eda-a025-f6b42562b1b7" containerID="d6ffebb59cd752a640d4f1cadc5008600f28c1264f0438708ff819197aa89125" exitCode=0 Jul 29 07:53:54.828577 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.828557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" event={"ID":"57a2a482-79f6-4eda-a025-f6b42562b1b7","Type":"ContainerDied","Data":"d6ffebb59cd752a640d4f1cadc5008600f28c1264f0438708ff819197aa89125"} Jul 29 07:53:54.831216 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.831138 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" Jul 29 07:53:54.831216 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.831033 2565 generic.go:356] "Generic (PLEG): container finished" podID="87b64fd9-e567-4a7f-8729-97320c956a65" containerID="f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351" exitCode=0 Jul 29 07:53:54.831216 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.831182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" event={"ID":"87b64fd9-e567-4a7f-8729-97320c956a65","Type":"ContainerDied","Data":"f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351"} Jul 29 07:53:54.833178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.831222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7" event={"ID":"87b64fd9-e567-4a7f-8729-97320c956a65","Type":"ContainerDied","Data":"5f53c63479cce47ab22c1897bc957a91a352daa8161cf2e60bc12d13c468e38c"} Jul 29 07:53:54.833178 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.831251 2565 scope.go:117] "RemoveContainer" containerID="f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351" Jul 29 07:53:54.843577 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.843558 2565 scope.go:117] "RemoveContainer" containerID="f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351" Jul 29 07:53:54.843934 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:53:54.843911 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351\": container with ID starting with f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351 not found: ID does not exist" containerID="f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351" Jul 29 07:53:54.844034 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.843948 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351"} err="failed to get container status \"f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351\": rpc error: code = NotFound desc = could not find container \"f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351\": container with ID starting with f561165fdea82aa75677c8fb334c1bc6b7da49bd57035cde5502c863b9687351 not found: ID does not exist" Jul 29 07:53:54.867965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.867922 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:53:54.878566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.878548 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:53:54.916552 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.916525 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbv2t\" (UniqueName: \"kubernetes.io/projected/87b64fd9-e567-4a7f-8729-97320c956a65-kube-api-access-wbv2t\") pod \"87b64fd9-e567-4a7f-8729-97320c956a65\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " Jul 29 07:53:54.916666 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.916623 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b64fd9-e567-4a7f-8729-97320c956a65-openshift-service-ca\") pod \"87b64fd9-e567-4a7f-8729-97320c956a65\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " Jul 29 07:53:54.916710 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.916688 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b64fd9-e567-4a7f-8729-97320c956a65-maas-api-tls\") pod \"87b64fd9-e567-4a7f-8729-97320c956a65\" (UID: \"87b64fd9-e567-4a7f-8729-97320c956a65\") " Jul 29 07:53:54.916977 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.916926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b64fd9-e567-4a7f-8729-97320c956a65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87b64fd9-e567-4a7f-8729-97320c956a65" (UID: "87b64fd9-e567-4a7f-8729-97320c956a65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:54.917094 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.917031 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b64fd9-e567-4a7f-8729-97320c956a65-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:54.918595 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.918575 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b64fd9-e567-4a7f-8729-97320c956a65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87b64fd9-e567-4a7f-8729-97320c956a65" (UID: "87b64fd9-e567-4a7f-8729-97320c956a65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:53:54.918679 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:54.918655 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b64fd9-e567-4a7f-8729-97320c956a65-kube-api-access-wbv2t" (OuterVolumeSpecName: "kube-api-access-wbv2t") pod "87b64fd9-e567-4a7f-8729-97320c956a65" (UID: "87b64fd9-e567-4a7f-8729-97320c956a65"). InnerVolumeSpecName "kube-api-access-wbv2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:55.017549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017455 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9bb568-6988-4330-a421-ea0fa9d27c04-plugins-config-volume\") pod \"2b9bb568-6988-4330-a421-ea0fa9d27c04\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " Jul 29 07:53:55.017549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5rq5\" (UniqueName: \"kubernetes.io/projected/2b9bb568-6988-4330-a421-ea0fa9d27c04-kube-api-access-s5rq5\") pod \"2b9bb568-6988-4330-a421-ea0fa9d27c04\" (UID: \"2b9bb568-6988-4330-a421-ea0fa9d27c04\") " Jul 29 07:53:55.017549 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017537 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2a482-79f6-4eda-a025-f6b42562b1b7-plugins-config-volume\") pod \"57a2a482-79f6-4eda-a025-f6b42562b1b7\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " Jul 29 07:53:55.017889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4nbp\" (UniqueName: \"kubernetes.io/projected/57a2a482-79f6-4eda-a025-f6b42562b1b7-kube-api-access-j4nbp\") pod \"57a2a482-79f6-4eda-a025-f6b42562b1b7\" (UID: \"57a2a482-79f6-4eda-a025-f6b42562b1b7\") " Jul 29 07:53:55.017889 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017861 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9bb568-6988-4330-a421-ea0fa9d27c04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b9bb568-6988-4330-a421-ea0fa9d27c04" (UID: "2b9bb568-6988-4330-a421-ea0fa9d27c04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:55.018024 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017911 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a2a482-79f6-4eda-a025-f6b42562b1b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57a2a482-79f6-4eda-a025-f6b42562b1b7" (UID: "57a2a482-79f6-4eda-a025-f6b42562b1b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:53:55.018024 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.017980 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbv2t\" (UniqueName: \"kubernetes.io/projected/87b64fd9-e567-4a7f-8729-97320c956a65-kube-api-access-wbv2t\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:55.018024 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.018008 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b64fd9-e567-4a7f-8729-97320c956a65-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:55.019687 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.019660 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9bb568-6988-4330-a421-ea0fa9d27c04-kube-api-access-s5rq5" (OuterVolumeSpecName: "kube-api-access-s5rq5") pod "2b9bb568-6988-4330-a421-ea0fa9d27c04" (UID: "2b9bb568-6988-4330-a421-ea0fa9d27c04"). InnerVolumeSpecName "kube-api-access-s5rq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:55.019800 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.019776 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a2a482-79f6-4eda-a025-f6b42562b1b7-kube-api-access-j4nbp" (OuterVolumeSpecName: "kube-api-access-j4nbp") pod "57a2a482-79f6-4eda-a025-f6b42562b1b7" (UID: "57a2a482-79f6-4eda-a025-f6b42562b1b7"). InnerVolumeSpecName "kube-api-access-j4nbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:53:55.118773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.118744 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9bb568-6988-4330-a421-ea0fa9d27c04-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:55.118773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.118770 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rq5\" (UniqueName: \"kubernetes.io/projected/2b9bb568-6988-4330-a421-ea0fa9d27c04-kube-api-access-s5rq5\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:55.118773 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.118780 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2a482-79f6-4eda-a025-f6b42562b1b7-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:55.119010 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.118789 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4nbp\" (UniqueName: \"kubernetes.io/projected/57a2a482-79f6-4eda-a025-f6b42562b1b7-kube-api-access-j4nbp\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:53:55.153023 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.152986 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7"] Jul 29 07:53:55.157593 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.157560 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6451b-54fdb4757d-5wlh7"] Jul 29 07:53:55.837789 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.837753 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" Jul 29 07:53:55.837789 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.837778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8" event={"ID":"57a2a482-79f6-4eda-a025-f6b42562b1b7","Type":"ContainerDied","Data":"8f1560a4860b32c411a7c0cb46b7ca3f69379ef8b2164ee50b06b5f84c0c4c2b"} Jul 29 07:53:55.838317 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.837872 2565 scope.go:117] "RemoveContainer" containerID="d6ffebb59cd752a640d4f1cadc5008600f28c1264f0438708ff819197aa89125" Jul 29 07:53:55.840502 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.840470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" event={"ID":"2b9bb568-6988-4330-a421-ea0fa9d27c04","Type":"ContainerDied","Data":"21bec44e4bdfbc8dd0cc4aea7674df3fbeb2e17b56cd12ac39942e665a2f2990"} Jul 29 07:53:55.840607 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.840513 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49" Jul 29 07:53:55.851684 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.851663 2565 scope.go:117] "RemoveContainer" containerID="25b374661bc2301b278933824d8a20632a0a81c68ea225810bb6fbd68513ad9f" Jul 29 07:53:55.862871 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.862845 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49"] Jul 29 07:53:55.865576 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.865553 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d6451b-648955579b-nqr49"] Jul 29 07:53:55.880755 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.880734 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8"] Jul 29 07:53:55.889486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:55.889463 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6451b-6b67d655cb-nfbp8"] Jul 29 07:53:57.019617 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:57.019578 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9bb568-6988-4330-a421-ea0fa9d27c04" path="/var/lib/kubelet/pods/2b9bb568-6988-4330-a421-ea0fa9d27c04/volumes" Jul 29 07:53:57.020006 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:57.019937 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57a2a482-79f6-4eda-a025-f6b42562b1b7" path="/var/lib/kubelet/pods/57a2a482-79f6-4eda-a025-f6b42562b1b7/volumes" Jul 29 07:53:57.020304 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:53:57.020289 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b64fd9-e567-4a7f-8729-97320c956a65" path="/var/lib/kubelet/pods/87b64fd9-e567-4a7f-8729-97320c956a65/volumes" Jul 29 07:54:07.305674 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.305639 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8"] Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306104 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9bb568-6988-4330-a421-ea0fa9d27c04" containerName="payload-pre-processing" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306126 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9bb568-6988-4330-a421-ea0fa9d27c04" containerName="payload-pre-processing" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306149 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6b2f14-944c-431b-8994-231d223e3eed" containerName="maas-api" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306154 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6b2f14-944c-431b-8994-231d223e3eed" containerName="maas-api" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306160 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a2a482-79f6-4eda-a025-f6b42562b1b7" containerName="payload-processing" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306166 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a2a482-79f6-4eda-a025-f6b42562b1b7" containerName="payload-processing" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306175 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c056237c-91f5-408d-9ed8-a22ca993ca0d" containerName="payload-pre-processing" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306180 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c056237c-91f5-408d-9ed8-a22ca993ca0d" containerName="payload-pre-processing" Jul 29 07:54:07.306186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306189 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07663c8-53e6-4de4-828f-25b8a5160265" containerName="payload-processing" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306194 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07663c8-53e6-4de4-828f-25b8a5160265" containerName="payload-processing" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306202 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b64fd9-e567-4a7f-8729-97320c956a65" containerName="maas-api" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306207 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b64fd9-e567-4a7f-8729-97320c956a65" containerName="maas-api" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306260 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07663c8-53e6-4de4-828f-25b8a5160265" containerName="payload-processing" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306268 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c056237c-91f5-408d-9ed8-a22ca993ca0d" containerName="payload-pre-processing" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306276 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b64fd9-e567-4a7f-8729-97320c956a65" containerName="maas-api" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306282 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a2a482-79f6-4eda-a025-f6b42562b1b7" containerName="payload-processing" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306289 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6b2f14-944c-431b-8994-231d223e3eed" containerName="maas-api" Jul 29 07:54:07.306521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.306294 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9bb568-6988-4330-a421-ea0fa9d27c04" containerName="payload-pre-processing" Jul 29 07:54:07.310927 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.310904 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.317314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.317283 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0d455e-serving-cert\"" Jul 29 07:54:07.317448 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.317334 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:54:07.318650 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.318634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:54:07.321783 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.321763 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:54:07.335873 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.335846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8"] Jul 29 07:54:07.436978 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.436914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d358ed00-204c-4aef-add3-009d037bcdfa-openshift-service-ca\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.437154 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.436999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhkq\" (UniqueName: \"kubernetes.io/projected/d358ed00-204c-4aef-add3-009d037bcdfa-kube-api-access-hhhkq\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.437154 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.437124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.537825 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.537790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhkq\" (UniqueName: \"kubernetes.io/projected/d358ed00-204c-4aef-add3-009d037bcdfa-kube-api-access-hhhkq\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.538202 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.537862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.538202 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.537916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d358ed00-204c-4aef-add3-009d037bcdfa-openshift-service-ca\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.538202 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:54:07.537983 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-serving-cert: secret "maas-api-e2e-api-a-0d455e-serving-cert" not found Jul 29 07:54:07.538202 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:54:07.538057 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls podName:d358ed00-204c-4aef-add3-009d037bcdfa nodeName:}" failed. No retries permitted until 2026-07-29 07:54:08.038039192 +0000 UTC m=+2329.644217279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls") pod "maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" (UID: "d358ed00-204c-4aef-add3-009d037bcdfa") : secret "maas-api-e2e-api-a-0d455e-serving-cert" not found Jul 29 07:54:07.538509 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.538492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d358ed00-204c-4aef-add3-009d037bcdfa-openshift-service-ca\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:07.560559 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:07.560498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhkq\" (UniqueName: \"kubernetes.io/projected/d358ed00-204c-4aef-add3-009d037bcdfa-kube-api-access-hhhkq\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:08.042672 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.042587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:08.045042 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.045021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls\") pod \"maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:08.221935 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.221902 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:08.343149 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.343112 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm"] Jul 29 07:54:08.347936 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.347911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.355138 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.355113 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0d455e-dockercfg-vhbkz\"" Jul 29 07:54:08.355255 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.355234 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0d455e\"" Jul 29 07:54:08.391068 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.391042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm"] Jul 29 07:54:08.393308 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.393267 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8"] Jul 29 07:54:08.396114 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.396080 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx"] Jul 29 07:54:08.399562 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:54:08.399537 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd358ed00_204c_4aef_add3_009d037bcdfa.slice/crio-250383ce7f46cdebacfe754bba47c7299ab544fa44b08edc7719f5a332708742 WatchSource:0}: Error finding container 250383ce7f46cdebacfe754bba47c7299ab544fa44b08edc7719f5a332708742: Status 404 returned error can't find the container with id 250383ce7f46cdebacfe754bba47c7299ab544fa44b08edc7719f5a332708742 Jul 29 07:54:08.410036 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.410017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.445710 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.445678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjxd\" (UniqueName: \"kubernetes.io/projected/7c5ffaa4-9279-4617-8f31-8168a061717f-kube-api-access-snjxd\") pod \"payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.445892 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.445737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5ffaa4-9279-4617-8f31-8168a061717f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.490722 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.490688 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx"] Jul 29 07:54:08.546475 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.546445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khplg\" (UniqueName: \"kubernetes.io/projected/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-kube-api-access-khplg\") pod \"payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.546634 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.546490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjxd\" (UniqueName: \"kubernetes.io/projected/7c5ffaa4-9279-4617-8f31-8168a061717f-kube-api-access-snjxd\") pod \"payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.546634 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.546524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.546634 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.546556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5ffaa4-9279-4617-8f31-8168a061717f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.547645 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.547616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5ffaa4-9279-4617-8f31-8168a061717f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.571464 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.571399 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjxd\" (UniqueName: \"kubernetes.io/projected/7c5ffaa4-9279-4617-8f31-8168a061717f-kube-api-access-snjxd\") pod \"payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.647107 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.647070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khplg\" (UniqueName: \"kubernetes.io/projected/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-kube-api-access-khplg\") pod \"payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.647271 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.647130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.648071 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.648045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.658111 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.658087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:08.661356 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.661317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khplg\" (UniqueName: \"kubernetes.io/projected/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-kube-api-access-khplg\") pod \"payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.723791 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.723757 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:08.912617 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:08.912590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" event={"ID":"d358ed00-204c-4aef-add3-009d037bcdfa","Type":"ContainerStarted","Data":"250383ce7f46cdebacfe754bba47c7299ab544fa44b08edc7719f5a332708742"} Jul 29 07:54:09.016145 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:54:09.016115 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5ffaa4_9279_4617_8f31_8168a061717f.slice/crio-e55dcdf9581d6c396d9e82d4104202363ecb001f143df84ba068fec17731457a WatchSource:0}: Error finding container e55dcdf9581d6c396d9e82d4104202363ecb001f143df84ba068fec17731457a: Status 404 returned error can't find the container with id e55dcdf9581d6c396d9e82d4104202363ecb001f143df84ba068fec17731457a Jul 29 07:54:09.019704 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.019680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm"] Jul 29 07:54:09.089811 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.089786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx"] Jul 29 07:54:09.090467 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:54:09.090445 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce10051_346d_4d4e_9ab8_9a22e0f5e734.slice/crio-6d5832ab5c10f622d6f4e74452434c4472f3d7607bcfbd7bf342b1ff8745f7f1 WatchSource:0}: Error finding container 6d5832ab5c10f622d6f4e74452434c4472f3d7607bcfbd7bf342b1ff8745f7f1: Status 404 returned error can't find the container with id 6d5832ab5c10f622d6f4e74452434c4472f3d7607bcfbd7bf342b1ff8745f7f1 Jul 29 07:54:09.920227 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.920187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" event={"ID":"d358ed00-204c-4aef-add3-009d037bcdfa","Type":"ContainerStarted","Data":"07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa"} Jul 29 07:54:09.920677 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.920301 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:09.921712 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.921690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" event={"ID":"7c5ffaa4-9279-4617-8f31-8168a061717f","Type":"ContainerStarted","Data":"342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad"} Jul 29 07:54:09.921855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.921735 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:09.921855 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.921758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" event={"ID":"7c5ffaa4-9279-4617-8f31-8168a061717f","Type":"ContainerStarted","Data":"e55dcdf9581d6c396d9e82d4104202363ecb001f143df84ba068fec17731457a"} Jul 29 07:54:09.923071 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.923047 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" event={"ID":"8ce10051-346d-4d4e-9ab8-9a22e0f5e734","Type":"ContainerStarted","Data":"43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0"} Jul 29 07:54:09.923170 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.923090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" event={"ID":"8ce10051-346d-4d4e-9ab8-9a22e0f5e734","Type":"ContainerStarted","Data":"6d5832ab5c10f622d6f4e74452434c4472f3d7607bcfbd7bf342b1ff8745f7f1"} Jul 29 07:54:09.923170 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.923160 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:09.945110 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.945059 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" podStartSLOduration=2.484244313 podStartE2EDuration="2.945046977s" totalImagesPullingTime="460.802664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:09.944285492 +0000 UTC m=+2331.550463603" watchObservedRunningTime="2026-07-29 07:54:09.945046977 +0000 UTC m=+2331.551225087" Jul 29 07:54:09.962435 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.962399 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" podStartSLOduration=1.96238627 podStartE2EDuration="1.96238627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:09.961669348 +0000 UTC m=+2331.567847457" watchObservedRunningTime="2026-07-29 07:54:09.96238627 +0000 UTC m=+2331.568564378" Jul 29 07:54:09.981025 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:09.980984 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" podStartSLOduration=1.980972797 podStartE2EDuration="1.980972797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:09.978492672 +0000 UTC m=+2331.584670782" watchObservedRunningTime="2026-07-29 07:54:09.980972797 +0000 UTC m=+2331.587150896" Jul 29 07:54:15.933515 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:15.933439 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:54:20.930132 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:20.930099 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:54:20.930651 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:20.930217 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:54:36.203026 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.202988 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z"] Jul 29 07:54:36.206546 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.206527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.209664 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.209643 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fd8a8c-serving-cert\"" Jul 29 07:54:36.219075 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.219054 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z"] Jul 29 07:54:36.285970 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.285932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78db3-1d65-4f41-91df-98a3aaad6532-maas-api-tls\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.286085 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.286017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgk8w\" (UniqueName: \"kubernetes.io/projected/aae78db3-1d65-4f41-91df-98a3aaad6532-kube-api-access-vgk8w\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.286085 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.286046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78db3-1d65-4f41-91df-98a3aaad6532-openshift-service-ca\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.386748 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.386714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78db3-1d65-4f41-91df-98a3aaad6532-maas-api-tls\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.386923 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.386762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgk8w\" (UniqueName: \"kubernetes.io/projected/aae78db3-1d65-4f41-91df-98a3aaad6532-kube-api-access-vgk8w\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.386923 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.386892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78db3-1d65-4f41-91df-98a3aaad6532-openshift-service-ca\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.387521 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.387498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78db3-1d65-4f41-91df-98a3aaad6532-openshift-service-ca\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.389319 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.389293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78db3-1d65-4f41-91df-98a3aaad6532-maas-api-tls\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.397125 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.397099 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgk8w\" (UniqueName: \"kubernetes.io/projected/aae78db3-1d65-4f41-91df-98a3aaad6532-kube-api-access-vgk8w\") pod \"maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.518844 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.518781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:36.661815 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.661776 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z"] Jul 29 07:54:36.663109 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:54:36.663081 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae78db3_1d65_4f41_91df_98a3aaad6532.slice/crio-d443f27220d8dd09132b4aab62077136bc8ef1e651d0dc0e39901bb82b7e402e WatchSource:0}: Error finding container d443f27220d8dd09132b4aab62077136bc8ef1e651d0dc0e39901bb82b7e402e: Status 404 returned error can't find the container with id d443f27220d8dd09132b4aab62077136bc8ef1e651d0dc0e39901bb82b7e402e Jul 29 07:54:36.966615 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.966582 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4"] Jul 29 07:54:36.971205 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.971185 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb"] Jul 29 07:54:36.971327 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.971296 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:36.974498 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.974480 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:36.974806 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.974782 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fd8a8c-dockercfg-q8m7x\"" Jul 29 07:54:36.974912 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.974785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fd8a8c\"" Jul 29 07:54:36.978615 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.978591 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4"] Jul 29 07:54:36.994067 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.993454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:36.994067 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.993519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x9bw\" (UniqueName: \"kubernetes.io/projected/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-kube-api-access-8x9bw\") pod \"payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:36.994067 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.993600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3c59c-2532-4325-83a7-492bf79ceac3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:36.994067 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.993629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7rx\" (UniqueName: \"kubernetes.io/projected/43f3c59c-2532-4325-83a7-492bf79ceac3-kube-api-access-9q7rx\") pod \"payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:36.995534 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:36.995513 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb"] Jul 29 07:54:37.070655 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.070614 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" event={"ID":"aae78db3-1d65-4f41-91df-98a3aaad6532","Type":"ContainerStarted","Data":"d443f27220d8dd09132b4aab62077136bc8ef1e651d0dc0e39901bb82b7e402e"} Jul 29 07:54:37.094362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.094323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3c59c-2532-4325-83a7-492bf79ceac3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:37.094564 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.094370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7rx\" (UniqueName: \"kubernetes.io/projected/43f3c59c-2532-4325-83a7-492bf79ceac3-kube-api-access-9q7rx\") pod \"payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:37.094564 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.094430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:37.094564 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.094488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x9bw\" (UniqueName: \"kubernetes.io/projected/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-kube-api-access-8x9bw\") pod \"payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:37.095252 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.095230 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3c59c-2532-4325-83a7-492bf79ceac3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:37.095347 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.095328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:37.105449 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.105421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7rx\" (UniqueName: \"kubernetes.io/projected/43f3c59c-2532-4325-83a7-492bf79ceac3-kube-api-access-9q7rx\") pod \"payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:37.105726 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.105708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x9bw\" (UniqueName: \"kubernetes.io/projected/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-kube-api-access-8x9bw\") pod \"payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:37.285086 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.285022 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:37.290760 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.290738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:37.645640 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.645612 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4"] Jul 29 07:54:37.646538 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:54:37.646515 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34c03cf_dee7_4af0_98f8_326eeb6e0eb6.slice/crio-ea102fac4515c75fe42b815201e1c1ba74a285e52724eaf50bb2c1907b1e0d20 WatchSource:0}: Error finding container ea102fac4515c75fe42b815201e1c1ba74a285e52724eaf50bb2c1907b1e0d20: Status 404 returned error can't find the container with id ea102fac4515c75fe42b815201e1c1ba74a285e52724eaf50bb2c1907b1e0d20 Jul 29 07:54:37.674469 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:37.674448 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb"] Jul 29 07:54:37.675687 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:54:37.675662 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f3c59c_2532_4325_83a7_492bf79ceac3.slice/crio-54d68955f73cc4ec092e796746d8a13429f9d769fc8f6396acb9b17539835beb WatchSource:0}: Error finding container 54d68955f73cc4ec092e796746d8a13429f9d769fc8f6396acb9b17539835beb: Status 404 returned error can't find the container with id 54d68955f73cc4ec092e796746d8a13429f9d769fc8f6396acb9b17539835beb Jul 29 07:54:38.078186 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.078149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" event={"ID":"43f3c59c-2532-4325-83a7-492bf79ceac3","Type":"ContainerStarted","Data":"f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9"} Jul 29 07:54:38.078569 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.078207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" event={"ID":"43f3c59c-2532-4325-83a7-492bf79ceac3","Type":"ContainerStarted","Data":"54d68955f73cc4ec092e796746d8a13429f9d769fc8f6396acb9b17539835beb"} Jul 29 07:54:38.078569 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.078262 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:38.079680 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.079657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" event={"ID":"aae78db3-1d65-4f41-91df-98a3aaad6532","Type":"ContainerStarted","Data":"054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f"} Jul 29 07:54:38.079801 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.079784 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:38.081245 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.081221 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" event={"ID":"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6","Type":"ContainerStarted","Data":"1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11"} Jul 29 07:54:38.081348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.081266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" event={"ID":"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6","Type":"ContainerStarted","Data":"ea102fac4515c75fe42b815201e1c1ba74a285e52724eaf50bb2c1907b1e0d20"} Jul 29 07:54:38.081348 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.081294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:54:38.111119 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.111081 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" podStartSLOduration=2.111068901 podStartE2EDuration="2.111068901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:38.109667318 +0000 UTC m=+2359.715845426" watchObservedRunningTime="2026-07-29 07:54:38.111068901 +0000 UTC m=+2359.717246987" Jul 29 07:54:38.171853 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.171800 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" podStartSLOduration=1.6905685099999999 podStartE2EDuration="2.171782055s" totalImagesPullingTime="481.213545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:38.171127033 +0000 UTC m=+2359.777305142" watchObservedRunningTime="2026-07-29 07:54:38.171782055 +0000 UTC m=+2359.777960166" Jul 29 07:54:38.172394 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:38.172362 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" podStartSLOduration=2.172351921 podStartE2EDuration="2.172351921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:54:38.139833785 +0000 UTC m=+2359.746011893" watchObservedRunningTime="2026-07-29 07:54:38.172351921 +0000 UTC m=+2359.778530031" Jul 29 07:54:44.092862 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:44.092830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:54:49.089742 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:49.089710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:54:49.090170 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:54:49.089842 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:55:19.905077 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:19.905043 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8"] Jul 29 07:55:19.905625 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:19.905348 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" podUID="d358ed00-204c-4aef-add3-009d037bcdfa" containerName="maas-api" containerID="cri-o://07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa" gracePeriod=30 Jul 29 07:55:19.959661 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:19.959616 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm"] Jul 29 07:55:19.960526 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:19.959929 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" podUID="7c5ffaa4-9279-4617-8f31-8168a061717f" containerName="payload-processing" containerID="cri-o://342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad" gracePeriod=30 Jul 29 07:55:19.964766 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:19.964740 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx"] Jul 29 07:55:19.965068 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:19.965005 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" podUID="8ce10051-346d-4d4e-9ab8-9a22e0f5e734" containerName="payload-pre-processing" containerID="cri-o://43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0" gracePeriod=30 Jul 29 07:55:20.245846 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.245820 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:55:20.249328 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.249306 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:55:20.255379 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.255363 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:55:20.304642 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.304615 2565 generic.go:356] "Generic (PLEG): container finished" podID="d358ed00-204c-4aef-add3-009d037bcdfa" containerID="07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa" exitCode=0 Jul 29 07:55:20.304782 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.304643 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" Jul 29 07:55:20.304782 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.304689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" event={"ID":"d358ed00-204c-4aef-add3-009d037bcdfa","Type":"ContainerDied","Data":"07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa"} Jul 29 07:55:20.304782 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.304734 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8" event={"ID":"d358ed00-204c-4aef-add3-009d037bcdfa","Type":"ContainerDied","Data":"250383ce7f46cdebacfe754bba47c7299ab544fa44b08edc7719f5a332708742"} Jul 29 07:55:20.304782 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.304761 2565 scope.go:117] "RemoveContainer" containerID="07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa" Jul 29 07:55:20.306078 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.306057 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c5ffaa4-9279-4617-8f31-8168a061717f" containerID="342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad" exitCode=0 Jul 29 07:55:20.306187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.306100 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" Jul 29 07:55:20.306187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.306135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" event={"ID":"7c5ffaa4-9279-4617-8f31-8168a061717f","Type":"ContainerDied","Data":"342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad"} Jul 29 07:55:20.306187 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.306167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm" event={"ID":"7c5ffaa4-9279-4617-8f31-8168a061717f","Type":"ContainerDied","Data":"e55dcdf9581d6c396d9e82d4104202363ecb001f143df84ba068fec17731457a"} Jul 29 07:55:20.307498 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.307477 2565 generic.go:356] "Generic (PLEG): container finished" podID="8ce10051-346d-4d4e-9ab8-9a22e0f5e734" containerID="43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0" exitCode=0 Jul 29 07:55:20.307579 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.307506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" event={"ID":"8ce10051-346d-4d4e-9ab8-9a22e0f5e734","Type":"ContainerDied","Data":"43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0"} Jul 29 07:55:20.307579 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.307517 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" Jul 29 07:55:20.307579 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.307548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx" event={"ID":"8ce10051-346d-4d4e-9ab8-9a22e0f5e734","Type":"ContainerDied","Data":"6d5832ab5c10f622d6f4e74452434c4472f3d7607bcfbd7bf342b1ff8745f7f1"} Jul 29 07:55:20.313862 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.313842 2565 scope.go:117] "RemoveContainer" containerID="07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa" Jul 29 07:55:20.314179 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:20.314156 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa\": container with ID starting with 07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa not found: ID does not exist" containerID="07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa" Jul 29 07:55:20.314272 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.314185 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa"} err="failed to get container status \"07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa\": rpc error: code = NotFound desc = could not find container \"07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa\": container with ID starting with 07f9f6d12a27d2ba57019a123d45d059d3e53b5c7353f31135cfe79fc843e2aa not found: ID does not exist" Jul 29 07:55:20.314272 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.314202 2565 scope.go:117] "RemoveContainer" containerID="342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad" Jul 29 07:55:20.321837 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.321819 2565 scope.go:117] "RemoveContainer" containerID="342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad" Jul 29 07:55:20.322174 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:20.322148 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad\": container with ID starting with 342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad not found: ID does not exist" containerID="342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad" Jul 29 07:55:20.322242 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.322179 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad"} err="failed to get container status \"342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad\": rpc error: code = NotFound desc = could not find container \"342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad\": container with ID starting with 342f4c90c11842f41c4b7c5e0f4963271cd3a94a18a8b33f7da77b773ca065ad not found: ID does not exist" Jul 29 07:55:20.322242 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.322195 2565 scope.go:117] "RemoveContainer" containerID="43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0" Jul 29 07:55:20.333309 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.333291 2565 scope.go:117] "RemoveContainer" containerID="43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0" Jul 29 07:55:20.333560 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:20.333541 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0\": container with ID starting with 43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0 not found: ID does not exist" containerID="43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0" Jul 29 07:55:20.333689 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.333565 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0"} err="failed to get container status \"43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0\": rpc error: code = NotFound desc = could not find container \"43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0\": container with ID starting with 43212473e252dc3ce8e05ba7151e22f80886722ec3a83667b357cd0920e460c0 not found: ID does not exist" Jul 29 07:55:20.379199 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379177 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d358ed00-204c-4aef-add3-009d037bcdfa-openshift-service-ca\") pod \"d358ed00-204c-4aef-add3-009d037bcdfa\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " Jul 29 07:55:20.379326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379211 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhhkq\" (UniqueName: \"kubernetes.io/projected/d358ed00-204c-4aef-add3-009d037bcdfa-kube-api-access-hhhkq\") pod \"d358ed00-204c-4aef-add3-009d037bcdfa\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " Jul 29 07:55:20.379326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379232 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5ffaa4-9279-4617-8f31-8168a061717f-plugins-config-volume\") pod \"7c5ffaa4-9279-4617-8f31-8168a061717f\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " Jul 29 07:55:20.379326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379267 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-plugins-config-volume\") pod \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " Jul 29 07:55:20.379326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snjxd\" (UniqueName: \"kubernetes.io/projected/7c5ffaa4-9279-4617-8f31-8168a061717f-kube-api-access-snjxd\") pod \"7c5ffaa4-9279-4617-8f31-8168a061717f\" (UID: \"7c5ffaa4-9279-4617-8f31-8168a061717f\") " Jul 29 07:55:20.379552 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379379 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls\") pod \"d358ed00-204c-4aef-add3-009d037bcdfa\" (UID: \"d358ed00-204c-4aef-add3-009d037bcdfa\") " Jul 29 07:55:20.379552 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379410 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khplg\" (UniqueName: \"kubernetes.io/projected/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-kube-api-access-khplg\") pod \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\" (UID: \"8ce10051-346d-4d4e-9ab8-9a22e0f5e734\") " Jul 29 07:55:20.379662 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379573 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d358ed00-204c-4aef-add3-009d037bcdfa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d358ed00-204c-4aef-add3-009d037bcdfa" (UID: "d358ed00-204c-4aef-add3-009d037bcdfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:20.379662 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379611 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5ffaa4-9279-4617-8f31-8168a061717f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c5ffaa4-9279-4617-8f31-8168a061717f" (UID: "7c5ffaa4-9279-4617-8f31-8168a061717f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:20.379662 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ce10051-346d-4d4e-9ab8-9a22e0f5e734" (UID: "8ce10051-346d-4d4e-9ab8-9a22e0f5e734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:20.379819 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379744 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d358ed00-204c-4aef-add3-009d037bcdfa-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.379819 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379764 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c5ffaa4-9279-4617-8f31-8168a061717f-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.379819 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.379779 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.381484 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.381457 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d358ed00-204c-4aef-add3-009d037bcdfa-kube-api-access-hhhkq" (OuterVolumeSpecName: "kube-api-access-hhhkq") pod "d358ed00-204c-4aef-add3-009d037bcdfa" (UID: "d358ed00-204c-4aef-add3-009d037bcdfa"). InnerVolumeSpecName "kube-api-access-hhhkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:20.381581 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.381492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d358ed00-204c-4aef-add3-009d037bcdfa" (UID: "d358ed00-204c-4aef-add3-009d037bcdfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:55:20.381775 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.381755 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5ffaa4-9279-4617-8f31-8168a061717f-kube-api-access-snjxd" (OuterVolumeSpecName: "kube-api-access-snjxd") pod "7c5ffaa4-9279-4617-8f31-8168a061717f" (UID: "7c5ffaa4-9279-4617-8f31-8168a061717f"). InnerVolumeSpecName "kube-api-access-snjxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:20.381858 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.381771 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-kube-api-access-khplg" (OuterVolumeSpecName: "kube-api-access-khplg") pod "8ce10051-346d-4d4e-9ab8-9a22e0f5e734" (UID: "8ce10051-346d-4d4e-9ab8-9a22e0f5e734"). InnerVolumeSpecName "kube-api-access-khplg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:20.481031 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.480972 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhhkq\" (UniqueName: \"kubernetes.io/projected/d358ed00-204c-4aef-add3-009d037bcdfa-kube-api-access-hhhkq\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.481031 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.480992 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snjxd\" (UniqueName: \"kubernetes.io/projected/7c5ffaa4-9279-4617-8f31-8168a061717f-kube-api-access-snjxd\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.481031 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.481004 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d358ed00-204c-4aef-add3-009d037bcdfa-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.481031 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.481013 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khplg\" (UniqueName: \"kubernetes.io/projected/8ce10051-346d-4d4e-9ab8-9a22e0f5e734-kube-api-access-khplg\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:20.631959 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.631932 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8"] Jul 29 07:55:20.636362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.636340 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d455e-86f8c8647-gqzv8"] Jul 29 07:55:20.650989 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.650969 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm"] Jul 29 07:55:20.657973 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.657939 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d455e-5f687f999d-6ksfm"] Jul 29 07:55:20.669851 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.669830 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx"] Jul 29 07:55:20.675888 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:20.675870 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d455e-7b7d64fd56-rqbgx"] Jul 29 07:55:21.019320 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:21.019293 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5ffaa4-9279-4617-8f31-8168a061717f" path="/var/lib/kubelet/pods/7c5ffaa4-9279-4617-8f31-8168a061717f/volumes" Jul 29 07:55:21.019666 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:21.019652 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce10051-346d-4d4e-9ab8-9a22e0f5e734" path="/var/lib/kubelet/pods/8ce10051-346d-4d4e-9ab8-9a22e0f5e734/volumes" Jul 29 07:55:21.020038 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:21.020025 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d358ed00-204c-4aef-add3-009d037bcdfa" path="/var/lib/kubelet/pods/d358ed00-204c-4aef-add3-009d037bcdfa/volumes" Jul 29 07:55:43.353404 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.353214 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z"] Jul 29 07:55:43.354288 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.354223 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" podUID="aae78db3-1d65-4f41-91df-98a3aaad6532" containerName="maas-api" containerID="cri-o://054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f" gracePeriod=30 Jul 29 07:55:43.435023 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.434988 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb"] Jul 29 07:55:43.435288 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.435228 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" podUID="43f3c59c-2532-4325-83a7-492bf79ceac3" containerName="payload-processing" containerID="cri-o://f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9" gracePeriod=30 Jul 29 07:55:43.443166 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.443142 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4"] Jul 29 07:55:43.443367 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.443325 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" podUID="b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" containerName="payload-pre-processing" containerID="cri-o://1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11" gracePeriod=30 Jul 29 07:55:43.711412 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.711387 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:55:43.727673 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.727652 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:55:43.738852 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.738827 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:55:43.769285 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78db3-1d65-4f41-91df-98a3aaad6532-maas-api-tls\") pod \"aae78db3-1d65-4f41-91df-98a3aaad6532\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " Jul 29 07:55:43.769431 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3c59c-2532-4325-83a7-492bf79ceac3-plugins-config-volume\") pod \"43f3c59c-2532-4325-83a7-492bf79ceac3\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " Jul 29 07:55:43.769431 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769378 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q7rx\" (UniqueName: \"kubernetes.io/projected/43f3c59c-2532-4325-83a7-492bf79ceac3-kube-api-access-9q7rx\") pod \"43f3c59c-2532-4325-83a7-492bf79ceac3\" (UID: \"43f3c59c-2532-4325-83a7-492bf79ceac3\") " Jul 29 07:55:43.769431 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769410 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78db3-1d65-4f41-91df-98a3aaad6532-openshift-service-ca\") pod \"aae78db3-1d65-4f41-91df-98a3aaad6532\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " Jul 29 07:55:43.769598 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgk8w\" (UniqueName: \"kubernetes.io/projected/aae78db3-1d65-4f41-91df-98a3aaad6532-kube-api-access-vgk8w\") pod \"aae78db3-1d65-4f41-91df-98a3aaad6532\" (UID: \"aae78db3-1d65-4f41-91df-98a3aaad6532\") " Jul 29 07:55:43.769767 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769739 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f3c59c-2532-4325-83a7-492bf79ceac3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43f3c59c-2532-4325-83a7-492bf79ceac3" (UID: "43f3c59c-2532-4325-83a7-492bf79ceac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:43.769841 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.769770 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae78db3-1d65-4f41-91df-98a3aaad6532-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aae78db3-1d65-4f41-91df-98a3aaad6532" (UID: "aae78db3-1d65-4f41-91df-98a3aaad6532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:43.771409 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.771380 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae78db3-1d65-4f41-91df-98a3aaad6532-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aae78db3-1d65-4f41-91df-98a3aaad6532" (UID: "aae78db3-1d65-4f41-91df-98a3aaad6532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 07:55:43.771510 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.771481 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f3c59c-2532-4325-83a7-492bf79ceac3-kube-api-access-9q7rx" (OuterVolumeSpecName: "kube-api-access-9q7rx") pod "43f3c59c-2532-4325-83a7-492bf79ceac3" (UID: "43f3c59c-2532-4325-83a7-492bf79ceac3"). InnerVolumeSpecName "kube-api-access-9q7rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:43.771567 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.771550 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae78db3-1d65-4f41-91df-98a3aaad6532-kube-api-access-vgk8w" (OuterVolumeSpecName: "kube-api-access-vgk8w") pod "aae78db3-1d65-4f41-91df-98a3aaad6532" (UID: "aae78db3-1d65-4f41-91df-98a3aaad6532"). InnerVolumeSpecName "kube-api-access-vgk8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:43.870122 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870031 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x9bw\" (UniqueName: \"kubernetes.io/projected/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-kube-api-access-8x9bw\") pod \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " Jul 29 07:55:43.870292 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870136 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-plugins-config-volume\") pod \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\" (UID: \"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6\") " Jul 29 07:55:43.870418 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870401 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7rx\" (UniqueName: \"kubernetes.io/projected/43f3c59c-2532-4325-83a7-492bf79ceac3-kube-api-access-9q7rx\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:43.870488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870424 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aae78db3-1d65-4f41-91df-98a3aaad6532-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:43.870488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870438 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgk8w\" (UniqueName: \"kubernetes.io/projected/aae78db3-1d65-4f41-91df-98a3aaad6532-kube-api-access-vgk8w\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:43.870488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870452 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aae78db3-1d65-4f41-91df-98a3aaad6532-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:43.870488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870453 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" (UID: "b34c03cf-dee7-4af0-98f8-326eeb6e0eb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 07:55:43.870488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.870465 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43f3c59c-2532-4325-83a7-492bf79ceac3-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:43.872053 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.872033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-kube-api-access-8x9bw" (OuterVolumeSpecName: "kube-api-access-8x9bw") pod "b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" (UID: "b34c03cf-dee7-4af0-98f8-326eeb6e0eb6"). InnerVolumeSpecName "kube-api-access-8x9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 07:55:43.971086 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.971061 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:43.971086 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:43.971082 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x9bw\" (UniqueName: \"kubernetes.io/projected/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6-kube-api-access-8x9bw\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 07:55:44.436012 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.435978 2565 generic.go:356] "Generic (PLEG): container finished" podID="43f3c59c-2532-4325-83a7-492bf79ceac3" containerID="f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9" exitCode=0 Jul 29 07:55:44.436460 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.436029 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" Jul 29 07:55:44.436460 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.436059 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" event={"ID":"43f3c59c-2532-4325-83a7-492bf79ceac3","Type":"ContainerDied","Data":"f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9"} Jul 29 07:55:44.436460 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.436117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb" event={"ID":"43f3c59c-2532-4325-83a7-492bf79ceac3","Type":"ContainerDied","Data":"54d68955f73cc4ec092e796746d8a13429f9d769fc8f6396acb9b17539835beb"} Jul 29 07:55:44.436460 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.436150 2565 scope.go:117] "RemoveContainer" containerID="f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9" Jul 29 07:55:44.437528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.437241 2565 generic.go:356] "Generic (PLEG): container finished" podID="aae78db3-1d65-4f41-91df-98a3aaad6532" containerID="054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f" exitCode=0 Jul 29 07:55:44.437528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.437289 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" Jul 29 07:55:44.437528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.437316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" event={"ID":"aae78db3-1d65-4f41-91df-98a3aaad6532","Type":"ContainerDied","Data":"054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f"} Jul 29 07:55:44.437528 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.437353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z" event={"ID":"aae78db3-1d65-4f41-91df-98a3aaad6532","Type":"ContainerDied","Data":"d443f27220d8dd09132b4aab62077136bc8ef1e651d0dc0e39901bb82b7e402e"} Jul 29 07:55:44.438782 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.438758 2565 generic.go:356] "Generic (PLEG): container finished" podID="b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" containerID="1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11" exitCode=0 Jul 29 07:55:44.438935 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.438794 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" Jul 29 07:55:44.438935 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.438811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" event={"ID":"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6","Type":"ContainerDied","Data":"1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11"} Jul 29 07:55:44.438935 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.438851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4" event={"ID":"b34c03cf-dee7-4af0-98f8-326eeb6e0eb6","Type":"ContainerDied","Data":"ea102fac4515c75fe42b815201e1c1ba74a285e52724eaf50bb2c1907b1e0d20"} Jul 29 07:55:44.446091 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.445930 2565 scope.go:117] "RemoveContainer" containerID="f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9" Jul 29 07:55:44.446276 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:44.446241 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9\": container with ID starting with f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9 not found: ID does not exist" containerID="f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9" Jul 29 07:55:44.446388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.446287 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9"} err="failed to get container status \"f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9\": rpc error: code = NotFound desc = could not find container \"f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9\": container with ID starting with f835d9f68c4919fa7b899cc77848e51598a2dd3c4271d16fe293ed9f52dedda9 not found: ID does not exist" Jul 29 07:55:44.446388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.446310 2565 scope.go:117] "RemoveContainer" containerID="054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f" Jul 29 07:55:44.454733 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.454717 2565 scope.go:117] "RemoveContainer" containerID="054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f" Jul 29 07:55:44.455005 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:44.454984 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f\": container with ID starting with 054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f not found: ID does not exist" containerID="054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f" Jul 29 07:55:44.455104 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.455010 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f"} err="failed to get container status \"054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f\": rpc error: code = NotFound desc = could not find container \"054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f\": container with ID starting with 054640734de1d8dd12a9c518c81e92e303ec32ada9b5f443b05988903415959f not found: ID does not exist" Jul 29 07:55:44.455104 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.455027 2565 scope.go:117] "RemoveContainer" containerID="1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11" Jul 29 07:55:44.462258 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.462239 2565 scope.go:117] "RemoveContainer" containerID="1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11" Jul 29 07:55:44.462485 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:44.462465 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11\": container with ID starting with 1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11 not found: ID does not exist" containerID="1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11" Jul 29 07:55:44.462552 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.462493 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11"} err="failed to get container status \"1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11\": rpc error: code = NotFound desc = could not find container \"1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11\": container with ID starting with 1ea3cec083080b9d2bcb8c77436f9f9c39bdd8131943c2b8b4d5c0b175b33c11 not found: ID does not exist" Jul 29 07:55:44.467136 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.467115 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb"] Jul 29 07:55:44.473077 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.473055 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd8a8c-84f7b596db-blbvb"] Jul 29 07:55:44.488561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.488536 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z"] Jul 29 07:55:44.492488 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.492471 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fd8a8c-9ff4bcd65-pwg4z"] Jul 29 07:55:44.509170 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.509151 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4"] Jul 29 07:55:44.513481 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:44.513462 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fd8a8c-5d4b7d7c6d-zbnc4"] Jul 29 07:55:45.019741 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:45.019707 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f3c59c-2532-4325-83a7-492bf79ceac3" path="/var/lib/kubelet/pods/43f3c59c-2532-4325-83a7-492bf79ceac3/volumes" Jul 29 07:55:45.020137 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:45.020124 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aae78db3-1d65-4f41-91df-98a3aaad6532" path="/var/lib/kubelet/pods/aae78db3-1d65-4f41-91df-98a3aaad6532/volumes" Jul 29 07:55:45.020439 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:45.020428 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" path="/var/lib/kubelet/pods/b34c03cf-dee7-4af0-98f8-326eeb6e0eb6/volumes" Jul 29 07:55:51.267880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.267849 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5"] Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268223 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f3c59c-2532-4325-83a7-492bf79ceac3" containerName="payload-processing" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268236 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f3c59c-2532-4325-83a7-492bf79ceac3" containerName="payload-processing" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268247 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" containerName="payload-pre-processing" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268254 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" containerName="payload-pre-processing" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268263 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d358ed00-204c-4aef-add3-009d037bcdfa" containerName="maas-api" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268270 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d358ed00-204c-4aef-add3-009d037bcdfa" containerName="maas-api" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268292 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae78db3-1d65-4f41-91df-98a3aaad6532" containerName="maas-api" Jul 29 07:55:51.268302 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268298 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae78db3-1d65-4f41-91df-98a3aaad6532" containerName="maas-api" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268309 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5ffaa4-9279-4617-8f31-8168a061717f" containerName="payload-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268314 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5ffaa4-9279-4617-8f31-8168a061717f" containerName="payload-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268321 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce10051-346d-4d4e-9ab8-9a22e0f5e734" containerName="payload-pre-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268326 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce10051-346d-4d4e-9ab8-9a22e0f5e734" containerName="payload-pre-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268393 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5ffaa4-9279-4617-8f31-8168a061717f" containerName="payload-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268402 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34c03cf-dee7-4af0-98f8-326eeb6e0eb6" containerName="payload-pre-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268409 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aae78db3-1d65-4f41-91df-98a3aaad6532" containerName="maas-api" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268415 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce10051-346d-4d4e-9ab8-9a22e0f5e734" containerName="payload-pre-processing" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268422 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d358ed00-204c-4aef-add3-009d037bcdfa" containerName="maas-api" Jul 29 07:55:51.268561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.268428 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f3c59c-2532-4325-83a7-492bf79ceac3" containerName="payload-processing" Jul 29 07:55:51.273342 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.273321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.276609 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.276591 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-984886-openshift-default-dockercfg-47t97\"" Jul 29 07:55:51.290405 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.290381 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5"] Jul 29 07:55:51.335362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-podinfo\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-token\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-credential-socket\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-envoy\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335473 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-data\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335648 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istiod-ca-cert\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335648 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5tzj\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-kube-api-access-f5tzj\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335648 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-certs\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.335753 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.335642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-socket\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.436731 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.436694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-token\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.436903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.436740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-credential-socket\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.436903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.436770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-envoy\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.436903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.436807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-data\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.436895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istiod-ca-cert\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.436948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5tzj\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-kube-api-access-f5tzj\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-certs\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437128 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-socket\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-podinfo\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437183 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-credential-socket\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437330 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-data\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-certs\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437503 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-socket\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.437599 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.437548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istiod-ca-cert\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.439080 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.439051 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-envoy\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.439351 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.439335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-podinfo\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.445701 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.445681 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-token\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.446134 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.446113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5tzj\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-kube-api-access-f5tzj\") pod \"e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.583914 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.583886 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:51.721362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.721330 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5"] Jul 29 07:55:51.722127 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:55:51.722102 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd1a8cc_6319_4b51_9bf1_07905a5131b9.slice/crio-983a55efb445d9ea5049384a0a4f66ac8b68fa4474a4a83ec1b88423aa38c53a WatchSource:0}: Error finding container 983a55efb445d9ea5049384a0a4f66ac8b68fa4474a4a83ec1b88423aa38c53a: Status 404 returned error can't find the container with id 983a55efb445d9ea5049384a0a4f66ac8b68fa4474a4a83ec1b88423aa38c53a Jul 29 07:55:51.724273 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.724238 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:55:51.724360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.724299 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:55:51.724360 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:51.724335 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:55:52.485935 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:52.485900 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" event={"ID":"afd1a8cc-6319-4b51-9bf1-07905a5131b9","Type":"ContainerStarted","Data":"05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b"} Jul 29 07:55:52.486425 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:52.485972 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" event={"ID":"afd1a8cc-6319-4b51-9bf1-07905a5131b9","Type":"ContainerStarted","Data":"983a55efb445d9ea5049384a0a4f66ac8b68fa4474a4a83ec1b88423aa38c53a"} Jul 29 07:55:52.519378 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:52.519319 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podStartSLOduration=1.519304478 podStartE2EDuration="1.519304478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:55:52.515581603 +0000 UTC m=+2434.121759723" watchObservedRunningTime="2026-07-29 07:55:52.519304478 +0000 UTC m=+2434.125482661" Jul 29 07:55:52.584631 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:52.584608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:52.589362 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:52.589342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:53.490923 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:53.490890 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:53.491891 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:53.491874 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:55:54.730196 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.730167 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs"] Jul 29 07:55:54.733536 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.733517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.736669 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.736645 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 07:55:54.738155 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.738135 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 07:55:54.738155 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.738152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t9lbj\"" Jul 29 07:55:54.738317 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.738149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-984886-serving-cert\"" Jul 29 07:55:54.744584 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.744564 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs"] Jul 29 07:55:54.866518 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.866480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c08e8-d431-4131-9467-9cdaa8dc53cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.866683 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.866536 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzp46\" (UniqueName: \"kubernetes.io/projected/b31c08e8-d431-4131-9467-9cdaa8dc53cd-kube-api-access-jzp46\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.866683 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.866671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.967767 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.967731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.967903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.967779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c08e8-d431-4131-9467-9cdaa8dc53cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.967903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.967814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzp46\" (UniqueName: \"kubernetes.io/projected/b31c08e8-d431-4131-9467-9cdaa8dc53cd-kube-api-access-jzp46\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.967903 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:54.967886 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-serving-cert: secret "maas-api-e2e-shared-a-984886-serving-cert" not found Jul 29 07:55:54.968055 ip-10-0-143-226 kubenswrapper[2565]: E0729 07:55:54.967948 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls podName:b31c08e8-d431-4131-9467-9cdaa8dc53cd nodeName:}" failed. No retries permitted until 2026-07-29 07:55:55.467931744 +0000 UTC m=+2437.074109830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls") pod "maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" (UID: "b31c08e8-d431-4131-9467-9cdaa8dc53cd") : secret "maas-api-e2e-shared-a-984886-serving-cert" not found Jul 29 07:55:54.968495 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.968475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c08e8-d431-4131-9467-9cdaa8dc53cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:54.978622 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:54.978588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzp46\" (UniqueName: \"kubernetes.io/projected/b31c08e8-d431-4131-9467-9cdaa8dc53cd-kube-api-access-jzp46\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:55.456283 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.456236 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw"] Jul 29 07:55:55.459662 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.459648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.462414 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.462390 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-984886\"" Jul 29 07:55:55.470561 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.470534 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw"] Jul 29 07:55:55.472470 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.472447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:55.474903 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.474884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:55.483699 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.483676 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn"] Jul 29 07:55:55.487647 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.487628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.490865 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.490846 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-984886-dockercfg-99shr\"" Jul 29 07:55:55.496051 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.496023 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn"] Jul 29 07:55:55.573840 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.573805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/2d572a38-158e-4370-973a-e635f191f0f6-kube-api-access-pdsl7\") pod \"payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.574041 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.573874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d572a38-158e-4370-973a-e635f191f0f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.574041 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.573995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c7q\" (UniqueName: \"kubernetes.io/projected/fd383e36-3f7c-4ecb-9032-7b752633533f-kube-api-access-86c7q\") pod \"payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.574199 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.574076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd383e36-3f7c-4ecb-9032-7b752633533f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.645265 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.645228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:55.674669 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.674640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c7q\" (UniqueName: \"kubernetes.io/projected/fd383e36-3f7c-4ecb-9032-7b752633533f-kube-api-access-86c7q\") pod \"payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.674816 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.674712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd383e36-3f7c-4ecb-9032-7b752633533f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.674816 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.674804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/2d572a38-158e-4370-973a-e635f191f0f6-kube-api-access-pdsl7\") pod \"payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.674928 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.674847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d572a38-158e-4370-973a-e635f191f0f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.675566 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.675533 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd383e36-3f7c-4ecb-9032-7b752633533f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.675814 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.675793 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d572a38-158e-4370-973a-e635f191f0f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.683388 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.683354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/2d572a38-158e-4370-973a-e635f191f0f6-kube-api-access-pdsl7\") pod \"payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.683525 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.683421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c7q\" (UniqueName: \"kubernetes.io/projected/fd383e36-3f7c-4ecb-9032-7b752633533f-kube-api-access-86c7q\") pod \"payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:55.772448 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.772418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:55.783649 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.783626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs"] Jul 29 07:55:55.787729 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:55:55.787703 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31c08e8_d431_4131_9467_9cdaa8dc53cd.slice/crio-c546a2713ad4f1e90b8b3331468ec8e0e463ec741099200b1168660bb1b2ec32 WatchSource:0}: Error finding container c546a2713ad4f1e90b8b3331468ec8e0e463ec741099200b1168660bb1b2ec32: Status 404 returned error can't find the container with id c546a2713ad4f1e90b8b3331468ec8e0e463ec741099200b1168660bb1b2ec32 Jul 29 07:55:55.789623 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.789604 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 07:55:55.799439 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:55.799412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:56.118200 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.118175 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw"] Jul 29 07:55:56.119319 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:55:56.119291 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d572a38_158e_4370_973a_e635f191f0f6.slice/crio-e3a6c62645fa778e39271916d6e25bbe6143b86c4f9830e0aad2c47f0ce3fc78 WatchSource:0}: Error finding container e3a6c62645fa778e39271916d6e25bbe6143b86c4f9830e0aad2c47f0ce3fc78: Status 404 returned error can't find the container with id e3a6c62645fa778e39271916d6e25bbe6143b86c4f9830e0aad2c47f0ce3fc78 Jul 29 07:55:56.149486 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.149465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn"] Jul 29 07:55:56.155005 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:55:56.154974 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd383e36_3f7c_4ecb_9032_7b752633533f.slice/crio-f37c9c309399f5f49f6aa00b5e71b3debdb20d4ee534e9bce46467d2f99c489b WatchSource:0}: Error finding container f37c9c309399f5f49f6aa00b5e71b3debdb20d4ee534e9bce46467d2f99c489b: Status 404 returned error can't find the container with id f37c9c309399f5f49f6aa00b5e71b3debdb20d4ee534e9bce46467d2f99c489b Jul 29 07:55:56.508723 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.508638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" event={"ID":"2d572a38-158e-4370-973a-e635f191f0f6","Type":"ContainerStarted","Data":"1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e"} Jul 29 07:55:56.508723 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.508713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" event={"ID":"2d572a38-158e-4370-973a-e635f191f0f6","Type":"ContainerStarted","Data":"e3a6c62645fa778e39271916d6e25bbe6143b86c4f9830e0aad2c47f0ce3fc78"} Jul 29 07:55:56.508929 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.508740 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:55:56.510325 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.510296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" event={"ID":"b31c08e8-d431-4131-9467-9cdaa8dc53cd","Type":"ContainerStarted","Data":"e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a"} Jul 29 07:55:56.510453 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.510341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" event={"ID":"b31c08e8-d431-4131-9467-9cdaa8dc53cd","Type":"ContainerStarted","Data":"c546a2713ad4f1e90b8b3331468ec8e0e463ec741099200b1168660bb1b2ec32"} Jul 29 07:55:56.510453 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.510383 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:55:56.511755 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.511735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" event={"ID":"fd383e36-3f7c-4ecb-9032-7b752633533f","Type":"ContainerStarted","Data":"983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462"} Jul 29 07:55:56.511816 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.511770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" event={"ID":"fd383e36-3f7c-4ecb-9032-7b752633533f","Type":"ContainerStarted","Data":"f37c9c309399f5f49f6aa00b5e71b3debdb20d4ee534e9bce46467d2f99c489b"} Jul 29 07:55:56.511816 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.511789 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:55:56.528303 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.528266 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" podStartSLOduration=1.5282547260000001 podStartE2EDuration="1.528254726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:55:56.524695922 +0000 UTC m=+2438.130874043" watchObservedRunningTime="2026-07-29 07:55:56.528254726 +0000 UTC m=+2438.134432833" Jul 29 07:55:56.545052 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.545001 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" podStartSLOduration=2.112769245 podStartE2EDuration="2.544985108s" totalImagesPullingTime="432.215863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:55:56.544431999 +0000 UTC m=+2438.150610107" watchObservedRunningTime="2026-07-29 07:55:56.544985108 +0000 UTC m=+2438.151163214" Jul 29 07:55:56.567179 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:55:56.567142 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" podStartSLOduration=1.567130787 podStartE2EDuration="1.567130787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:55:56.56538606 +0000 UTC m=+2438.171564180" watchObservedRunningTime="2026-07-29 07:55:56.567130787 +0000 UTC m=+2438.173308895" Jul 29 07:56:02.522965 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:02.522916 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 07:56:07.519397 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:07.519367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 07:56:07.519807 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:07.519442 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 07:56:08.492398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.492324 2565 patch_prober.go:28] interesting pod/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 07:56:08.492398 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.492377 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 07:56:08.592314 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.592283 2565 generic.go:356] "Generic (PLEG): container finished" podID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerID="05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b" exitCode=0 Jul 29 07:56:08.592662 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.592356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" event={"ID":"afd1a8cc-6319-4b51-9bf1-07905a5131b9","Type":"ContainerDied","Data":"05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b"} Jul 29 07:56:08.592749 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.592736 2565 scope.go:117] "RemoveContainer" containerID="05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b" Jul 29 07:56:08.593755 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.593722 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:56:08.593896 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.593787 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:56:08.593896 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:08.593816 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 07:56:09.598852 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:09.598819 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" event={"ID":"afd1a8cc-6319-4b51-9bf1-07905a5131b9","Type":"ContainerStarted","Data":"fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507"} Jul 29 07:56:10.584879 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:10.584845 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:56:10.585115 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:10.585094 2565 patch_prober.go:28] interesting pod/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 07:56:10.585163 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:10.585140 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 07:56:11.584313 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:11.584270 2565 patch_prober.go:28] interesting pod/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 07:56:11.584789 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:11.584336 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 07:56:12.584972 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:12.584917 2565 patch_prober.go:28] interesting pod/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 29 07:56:12.585380 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:12.584996 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 29 07:56:13.588321 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:13.588284 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:56:13.588819 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:13.588650 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:56:13.589326 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:13.589304 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 07:56:36.906696 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.906664 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7"] Jul 29 07:56:36.909437 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.909421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:36.913743 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.913725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-71e038-dockercfg-99thr\"" Jul 29 07:56:36.913863 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.913823 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-71e038\"" Jul 29 07:56:36.938146 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.938121 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7"] Jul 29 07:56:36.947295 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.947263 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv"] Jul 29 07:56:36.949751 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.949729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:36.973577 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:36.973556 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv"] Jul 29 07:56:37.069378 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.069348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.069501 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.069452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99567592-6525-4c92-b72c-7150562c79ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.069501 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.069496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgnn\" (UniqueName: \"kubernetes.io/projected/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-kube-api-access-5fgnn\") pod \"payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.069587 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.069523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4hl\" (UniqueName: \"kubernetes.io/projected/99567592-6525-4c92-b72c-7150562c79ed-kube-api-access-qd4hl\") pod \"payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.170694 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.170630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4hl\" (UniqueName: \"kubernetes.io/projected/99567592-6525-4c92-b72c-7150562c79ed-kube-api-access-qd4hl\") pod \"payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.170694 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.170680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.170880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.170753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99567592-6525-4c92-b72c-7150562c79ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.170880 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.170776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgnn\" (UniqueName: \"kubernetes.io/projected/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-kube-api-access-5fgnn\") pod \"payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.171453 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.171434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.171497 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.171441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99567592-6525-4c92-b72c-7150562c79ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.182220 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.182190 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4hl\" (UniqueName: \"kubernetes.io/projected/99567592-6525-4c92-b72c-7150562c79ed-kube-api-access-qd4hl\") pod \"payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.182496 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.182474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgnn\" (UniqueName: \"kubernetes.io/projected/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-kube-api-access-5fgnn\") pod \"payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.219875 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.219856 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.260272 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.260246 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.392079 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.392049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7"] Jul 29 07:56:37.393115 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:56:37.393078 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d8e380_bc4b_4a1b_97d7_7fdcf0c50c8a.slice/crio-44643049f33b2618580f130a4ea7ee5dbca380ef4bc093e3af5f688ea2c59f2a WatchSource:0}: Error finding container 44643049f33b2618580f130a4ea7ee5dbca380ef4bc093e3af5f688ea2c59f2a: Status 404 returned error can't find the container with id 44643049f33b2618580f130a4ea7ee5dbca380ef4bc093e3af5f688ea2c59f2a Jul 29 07:56:37.447732 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.447709 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv"] Jul 29 07:56:37.448981 ip-10-0-143-226 kubenswrapper[2565]: W0729 07:56:37.448940 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99567592_6525_4c92_b72c_7150562c79ed.slice/crio-33aca1d5ff78b4cc1b8ff34262156d4e585d6c296f05c4f35cfaaefca44da856 WatchSource:0}: Error finding container 33aca1d5ff78b4cc1b8ff34262156d4e585d6c296f05c4f35cfaaefca44da856: Status 404 returned error can't find the container with id 33aca1d5ff78b4cc1b8ff34262156d4e585d6c296f05c4f35cfaaefca44da856 Jul 29 07:56:37.747407 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.747306 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" event={"ID":"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a","Type":"ContainerStarted","Data":"b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e"} Jul 29 07:56:37.747407 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.747378 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:37.747407 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.747405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" event={"ID":"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a","Type":"ContainerStarted","Data":"44643049f33b2618580f130a4ea7ee5dbca380ef4bc093e3af5f688ea2c59f2a"} Jul 29 07:56:37.749108 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.749078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" event={"ID":"99567592-6525-4c92-b72c-7150562c79ed","Type":"ContainerStarted","Data":"852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82"} Jul 29 07:56:37.749231 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.749132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" event={"ID":"99567592-6525-4c92-b72c-7150562c79ed","Type":"ContainerStarted","Data":"33aca1d5ff78b4cc1b8ff34262156d4e585d6c296f05c4f35cfaaefca44da856"} Jul 29 07:56:37.749392 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.749354 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:37.776881 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:37.776832 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" podStartSLOduration=1.776815448 podStartE2EDuration="1.776815448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:56:37.773635099 +0000 UTC m=+2479.379813207" watchObservedRunningTime="2026-07-29 07:56:37.776815448 +0000 UTC m=+2479.382993558" Jul 29 07:56:48.756793 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:48.756757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 07:56:48.757206 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:48.757073 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 07:56:48.776584 ip-10-0-143-226 kubenswrapper[2565]: I0729 07:56:48.776534 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" podStartSLOduration=12.776520028 podStartE2EDuration="12.776520028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:56:37.80474165 +0000 UTC m=+2479.410919757" watchObservedRunningTime="2026-07-29 07:56:48.776520028 +0000 UTC m=+2490.382698136" Jul 29 08:06:59.891923 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:06:59.891882 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd"] Jul 29 08:06:59.895398 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:06:59.895376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:06:59.900903 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:06:59.900886 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-940406-serving-cert\"" Jul 29 08:06:59.915499 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:06:59.915477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd"] Jul 29 08:07:00.037443 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.037412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbsc\" (UniqueName: \"kubernetes.io/projected/824ecf8b-a026-4873-925e-cf8a4fd71177-kube-api-access-xmbsc\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.037614 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.037562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824ecf8b-a026-4873-925e-cf8a4fd71177-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.037614 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.037600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.138206 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.138172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824ecf8b-a026-4873-925e-cf8a4fd71177-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.138372 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.138217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.138372 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.138248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbsc\" (UniqueName: \"kubernetes.io/projected/824ecf8b-a026-4873-925e-cf8a4fd71177-kube-api-access-xmbsc\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.138493 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:07:00.138377 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-serving-cert: secret "maas-api-e2e-ipp-a-940406-serving-cert" not found Jul 29 08:07:00.138493 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:07:00.138465 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls podName:824ecf8b-a026-4873-925e-cf8a4fd71177 nodeName:}" failed. No retries permitted until 2026-07-29 08:07:00.638447749 +0000 UTC m=+3102.244625835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls") pod "maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" (UID: "824ecf8b-a026-4873-925e-cf8a4fd71177") : secret "maas-api-e2e-ipp-a-940406-serving-cert" not found Jul 29 08:07:00.138847 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.138829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824ecf8b-a026-4873-925e-cf8a4fd71177-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.171128 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.171078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbsc\" (UniqueName: \"kubernetes.io/projected/824ecf8b-a026-4873-925e-cf8a4fd71177-kube-api-access-xmbsc\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.641333 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.641299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.643666 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.643641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls\") pod \"maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:00.807845 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:00.807810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:01.030157 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.030122 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p"] Jul 29 08:07:01.033911 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.033890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.037117 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.037094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-940406\"" Jul 29 08:07:01.052738 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.052713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p"] Jul 29 08:07:01.148048 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.147977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.148242 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.148183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfxc\" (UniqueName: \"kubernetes.io/projected/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-kube-api-access-7bfxc\") pod \"payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.176742 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.176713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd"] Jul 29 08:07:01.178892 ip-10-0-143-226 kubenswrapper[2565]: W0729 08:07:01.178863 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824ecf8b_a026_4873_925e_cf8a4fd71177.slice/crio-85e7b11f750ed713541276cadc1d04514054a62a64e5cf7451191e10b006ed37 WatchSource:0}: Error finding container 85e7b11f750ed713541276cadc1d04514054a62a64e5cf7451191e10b006ed37: Status 404 returned error can't find the container with id 85e7b11f750ed713541276cadc1d04514054a62a64e5cf7451191e10b006ed37 Jul 29 08:07:01.180246 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.180225 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:07:01.249581 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.249553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfxc\" (UniqueName: \"kubernetes.io/projected/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-kube-api-access-7bfxc\") pod \"payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.249679 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.249621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.250355 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.250337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.258655 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.258626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfxc\" (UniqueName: \"kubernetes.io/projected/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-kube-api-access-7bfxc\") pod \"payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.345384 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.345361 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:01.483259 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:01.483226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p"] Jul 29 08:07:01.484530 ip-10-0-143-226 kubenswrapper[2565]: W0729 08:07:01.484504 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b8ffc4_5ab4_431a_bb4c_aad906fab837.slice/crio-21be767d68f2d970f471e6e3d972232710744a9d0f4d2c3e83e104b5a9a04503 WatchSource:0}: Error finding container 21be767d68f2d970f471e6e3d972232710744a9d0f4d2c3e83e104b5a9a04503: Status 404 returned error can't find the container with id 21be767d68f2d970f471e6e3d972232710744a9d0f4d2c3e83e104b5a9a04503 Jul 29 08:07:02.095250 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.095216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" event={"ID":"b4b8ffc4-5ab4-431a-bb4c-aad906fab837","Type":"ContainerStarted","Data":"98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599"} Jul 29 08:07:02.095715 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.095272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" event={"ID":"b4b8ffc4-5ab4-431a-bb4c-aad906fab837","Type":"ContainerStarted","Data":"21be767d68f2d970f471e6e3d972232710744a9d0f4d2c3e83e104b5a9a04503"} Jul 29 08:07:02.095715 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.095301 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:02.096788 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.096764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" event={"ID":"824ecf8b-a026-4873-925e-cf8a4fd71177","Type":"ContainerStarted","Data":"2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709"} Jul 29 08:07:02.096875 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.096810 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:02.096875 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.096827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" event={"ID":"824ecf8b-a026-4873-925e-cf8a4fd71177","Type":"ContainerStarted","Data":"85e7b11f750ed713541276cadc1d04514054a62a64e5cf7451191e10b006ed37"} Jul 29 08:07:02.115862 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.115815 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" podStartSLOduration=1.115802181 podStartE2EDuration="1.115802181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:07:02.114592811 +0000 UTC m=+3103.720770920" watchObservedRunningTime="2026-07-29 08:07:02.115802181 +0000 UTC m=+3103.721980271" Jul 29 08:07:02.140666 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:02.140624 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" podStartSLOduration=2.642347062 podStartE2EDuration="3.140610606s" totalImagesPullingTime="498.263544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:07:02.137178205 +0000 UTC m=+3103.743356313" watchObservedRunningTime="2026-07-29 08:07:02.140610606 +0000 UTC m=+3103.746788714" Jul 29 08:07:08.108729 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:08.108702 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:07:13.104606 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:13.104576 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:07:21.578126 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.578091 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh"] Jul 29 08:07:21.583173 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.583150 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.587212 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.587193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f1dc98\"" Jul 29 08:07:21.587337 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.587316 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f1dc98-dockercfg-5wb2t\"" Jul 29 08:07:21.594307 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.594283 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh"] Jul 29 08:07:21.622757 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.622727 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c"] Jul 29 08:07:21.626270 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.626255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.631799 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.631773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.631912 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.631898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6txc\" (UniqueName: \"kubernetes.io/projected/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-kube-api-access-d6txc\") pod \"payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.637010 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.636967 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c"] Jul 29 08:07:21.733443 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.733409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqwh\" (UniqueName: \"kubernetes.io/projected/c18a6983-3874-4a51-8237-356b8f6cd8d8-kube-api-access-bgqwh\") pod \"payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.733443 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.733445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18a6983-3874-4a51-8237-356b8f6cd8d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.733732 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.733566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6txc\" (UniqueName: \"kubernetes.io/projected/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-kube-api-access-d6txc\") pod \"payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.733732 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.733611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.734446 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.734425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.742583 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.742557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6txc\" (UniqueName: \"kubernetes.io/projected/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-kube-api-access-d6txc\") pod \"payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.834134 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.834062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqwh\" (UniqueName: \"kubernetes.io/projected/c18a6983-3874-4a51-8237-356b8f6cd8d8-kube-api-access-bgqwh\") pod \"payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.834134 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.834096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18a6983-3874-4a51-8237-356b8f6cd8d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.834763 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.834746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18a6983-3874-4a51-8237-356b8f6cd8d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.843366 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.843339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqwh\" (UniqueName: \"kubernetes.io/projected/c18a6983-3874-4a51-8237-356b8f6cd8d8-kube-api-access-bgqwh\") pod \"payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:21.895059 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.895028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:21.937088 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:21.937057 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:22.045879 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.045845 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh"] Jul 29 08:07:22.047548 ip-10-0-143-226 kubenswrapper[2565]: W0729 08:07:22.047516 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bafdb2_154c_4f2a_ab49_ee26eb9ab80a.slice/crio-db8a56c2ebad7085e42708264f0bd61835c7ab12eb6c97409611c739429ae4c0 WatchSource:0}: Error finding container db8a56c2ebad7085e42708264f0bd61835c7ab12eb6c97409611c739429ae4c0: Status 404 returned error can't find the container with id db8a56c2ebad7085e42708264f0bd61835c7ab12eb6c97409611c739429ae4c0 Jul 29 08:07:22.090931 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.090909 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c"] Jul 29 08:07:22.096401 ip-10-0-143-226 kubenswrapper[2565]: W0729 08:07:22.096374 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18a6983_3874_4a51_8237_356b8f6cd8d8.slice/crio-bd50dad29f2cac8453e8eed7f430020ea412b71d8c48a3cdd7dc6eb285d828a5 WatchSource:0}: Error finding container bd50dad29f2cac8453e8eed7f430020ea412b71d8c48a3cdd7dc6eb285d828a5: Status 404 returned error can't find the container with id bd50dad29f2cac8453e8eed7f430020ea412b71d8c48a3cdd7dc6eb285d828a5 Jul 29 08:07:22.211031 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.211002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" event={"ID":"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a","Type":"ContainerStarted","Data":"5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6"} Jul 29 08:07:22.211184 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.211061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" event={"ID":"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a","Type":"ContainerStarted","Data":"db8a56c2ebad7085e42708264f0bd61835c7ab12eb6c97409611c739429ae4c0"} Jul 29 08:07:22.211184 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.211170 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:22.212653 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.212628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" event={"ID":"c18a6983-3874-4a51-8237-356b8f6cd8d8","Type":"ContainerStarted","Data":"19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724"} Jul 29 08:07:22.212773 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.212666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" event={"ID":"c18a6983-3874-4a51-8237-356b8f6cd8d8","Type":"ContainerStarted","Data":"bd50dad29f2cac8453e8eed7f430020ea412b71d8c48a3cdd7dc6eb285d828a5"} Jul 29 08:07:22.212773 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.212727 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:22.229662 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.229620 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" podStartSLOduration=1.229605527 podStartE2EDuration="1.229605527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:07:22.228521556 +0000 UTC m=+3123.834699704" watchObservedRunningTime="2026-07-29 08:07:22.229605527 +0000 UTC m=+3123.835783634" Jul 29 08:07:22.248011 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:22.247966 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" podStartSLOduration=1.247931799 podStartE2EDuration="1.247931799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:07:22.245341112 +0000 UTC m=+3123.851519221" watchObservedRunningTime="2026-07-29 08:07:22.247931799 +0000 UTC m=+3123.854109908" Jul 29 08:07:33.222491 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:33.222456 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:07:33.223059 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:33.223040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:07:46.002497 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:46.002421 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8npf2_855fe327-9db6-4515-a618-2ee91f801bc8/payload-processing/0.log" Jul 29 08:07:46.141929 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:07:46.141902 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh_19bafdb2-154c-4f2a-ab49-ee26eb9ab80a/payload-processing/0.log" Jul 29 08:08:38.614084 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:08:38.614053 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p_b4b8ffc4-5ab4-431a-bb4c-aad906fab837/payload-processing/0.log" Jul 29 08:08:38.757244 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:08:38.757215 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8npf2_855fe327-9db6-4515-a618-2ee91f801bc8/payload-processing/0.log" Jul 29 08:08:38.885944 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:08:38.885855 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh_19bafdb2-154c-4f2a-ab49-ee26eb9ab80a/payload-processing/0.log" Jul 29 08:09:36.091353 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.091266 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd"] Jul 29 08:09:36.092305 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.092268 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" podUID="824ecf8b-a026-4873-925e-cf8a4fd71177" containerName="maas-api" containerID="cri-o://2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709" gracePeriod=30 Jul 29 08:09:36.159553 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.159517 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p"] Jul 29 08:09:36.159781 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.159754 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" podUID="b4b8ffc4-5ab4-431a-bb4c-aad906fab837" containerName="payload-processing" containerID="cri-o://98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599" gracePeriod=30 Jul 29 08:09:36.434020 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.433992 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:09:36.472841 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.472806 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-plugins-config-volume\") pod \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " Jul 29 08:09:36.473034 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.472863 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bfxc\" (UniqueName: \"kubernetes.io/projected/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-kube-api-access-7bfxc\") pod \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\" (UID: \"b4b8ffc4-5ab4-431a-bb4c-aad906fab837\") " Jul 29 08:09:36.473257 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.473230 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4b8ffc4-5ab4-431a-bb4c-aad906fab837" (UID: "b4b8ffc4-5ab4-431a-bb4c-aad906fab837"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:09:36.475112 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.475083 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-kube-api-access-7bfxc" (OuterVolumeSpecName: "kube-api-access-7bfxc") pod "b4b8ffc4-5ab4-431a-bb4c-aad906fab837" (UID: "b4b8ffc4-5ab4-431a-bb4c-aad906fab837"). InnerVolumeSpecName "kube-api-access-7bfxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:09:36.565321 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.565297 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:09:36.573746 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.573726 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:09:36.573746 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.573748 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfxc\" (UniqueName: \"kubernetes.io/projected/b4b8ffc4-5ab4-431a-bb4c-aad906fab837-kube-api-access-7bfxc\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:09:36.675005 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.674897 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls\") pod \"824ecf8b-a026-4873-925e-cf8a4fd71177\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " Jul 29 08:09:36.675005 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.674992 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmbsc\" (UniqueName: \"kubernetes.io/projected/824ecf8b-a026-4873-925e-cf8a4fd71177-kube-api-access-xmbsc\") pod \"824ecf8b-a026-4873-925e-cf8a4fd71177\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " Jul 29 08:09:36.675254 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.675043 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824ecf8b-a026-4873-925e-cf8a4fd71177-openshift-service-ca\") pod \"824ecf8b-a026-4873-925e-cf8a4fd71177\" (UID: \"824ecf8b-a026-4873-925e-cf8a4fd71177\") " Jul 29 08:09:36.675414 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.675387 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824ecf8b-a026-4873-925e-cf8a4fd71177-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "824ecf8b-a026-4873-925e-cf8a4fd71177" (UID: "824ecf8b-a026-4873-925e-cf8a4fd71177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:09:36.676993 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.676929 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "824ecf8b-a026-4873-925e-cf8a4fd71177" (UID: "824ecf8b-a026-4873-925e-cf8a4fd71177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:09:36.676993 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.676945 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824ecf8b-a026-4873-925e-cf8a4fd71177-kube-api-access-xmbsc" (OuterVolumeSpecName: "kube-api-access-xmbsc") pod "824ecf8b-a026-4873-925e-cf8a4fd71177" (UID: "824ecf8b-a026-4873-925e-cf8a4fd71177"). InnerVolumeSpecName "kube-api-access-xmbsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:09:36.775927 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.775891 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824ecf8b-a026-4873-925e-cf8a4fd71177-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:09:36.775927 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.775921 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824ecf8b-a026-4873-925e-cf8a4fd71177-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:09:36.776143 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.775934 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbsc\" (UniqueName: \"kubernetes.io/projected/824ecf8b-a026-4873-925e-cf8a4fd71177-kube-api-access-xmbsc\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:09:36.972392 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.972306 2565 generic.go:356] "Generic (PLEG): container finished" podID="b4b8ffc4-5ab4-431a-bb4c-aad906fab837" containerID="98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599" exitCode=0 Jul 29 08:09:36.972392 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.972357 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" Jul 29 08:09:36.972392 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.972386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" event={"ID":"b4b8ffc4-5ab4-431a-bb4c-aad906fab837","Type":"ContainerDied","Data":"98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599"} Jul 29 08:09:36.972664 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.972438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p" event={"ID":"b4b8ffc4-5ab4-431a-bb4c-aad906fab837","Type":"ContainerDied","Data":"21be767d68f2d970f471e6e3d972232710744a9d0f4d2c3e83e104b5a9a04503"} Jul 29 08:09:36.972664 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.972463 2565 scope.go:117] "RemoveContainer" containerID="98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599" Jul 29 08:09:36.973769 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.973741 2565 generic.go:356] "Generic (PLEG): container finished" podID="824ecf8b-a026-4873-925e-cf8a4fd71177" containerID="2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709" exitCode=0 Jul 29 08:09:36.973883 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.973780 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" Jul 29 08:09:36.973883 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.973809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" event={"ID":"824ecf8b-a026-4873-925e-cf8a4fd71177","Type":"ContainerDied","Data":"2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709"} Jul 29 08:09:36.973883 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.973856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd" event={"ID":"824ecf8b-a026-4873-925e-cf8a4fd71177","Type":"ContainerDied","Data":"85e7b11f750ed713541276cadc1d04514054a62a64e5cf7451191e10b006ed37"} Jul 29 08:09:36.982586 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.982565 2565 scope.go:117] "RemoveContainer" containerID="98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599" Jul 29 08:09:36.982858 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:09:36.982827 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599\": container with ID starting with 98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599 not found: ID does not exist" containerID="98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599" Jul 29 08:09:36.982976 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.982857 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599"} err="failed to get container status \"98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599\": rpc error: code = NotFound desc = could not find container \"98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599\": container with ID starting with 98eafe89afa1d75cf5b2fc781f80c9a9023a9d0758053be2c6fd215491805599 not found: ID does not exist" Jul 29 08:09:36.982976 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.982874 2565 scope.go:117] "RemoveContainer" containerID="2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709" Jul 29 08:09:36.991043 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.991027 2565 scope.go:117] "RemoveContainer" containerID="2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709" Jul 29 08:09:36.991328 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:09:36.991303 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709\": container with ID starting with 2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709 not found: ID does not exist" containerID="2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709" Jul 29 08:09:36.991417 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:36.991336 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709"} err="failed to get container status \"2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709\": rpc error: code = NotFound desc = could not find container \"2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709\": container with ID starting with 2494b1468a2e4139dfa1827197ee873ceb2608ad830e76ed398be0a96687d709 not found: ID does not exist" Jul 29 08:09:37.004039 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:37.004014 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd"] Jul 29 08:09:37.008394 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:37.008372 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-940406-7f4d5f684d-b49rd"] Jul 29 08:09:37.019368 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:37.019321 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824ecf8b-a026-4873-925e-cf8a4fd71177" path="/var/lib/kubelet/pods/824ecf8b-a026-4873-925e-cf8a4fd71177/volumes" Jul 29 08:09:37.022816 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:37.022796 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p"] Jul 29 08:09:37.027310 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:37.027291 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-940406-5bcf459848-zwd2p"] Jul 29 08:09:39.020258 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:09:39.020224 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b8ffc4-5ab4-431a-bb4c-aad906fab837" path="/var/lib/kubelet/pods/b4b8ffc4-5ab4-431a-bb4c-aad906fab837/volumes" Jul 29 08:10:35.579482 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:35.579448 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh"] Jul 29 08:10:35.579977 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:35.579730 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" podUID="19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" containerName="payload-processing" containerID="cri-o://5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6" gracePeriod=30 Jul 29 08:10:35.594619 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:35.594593 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c"] Jul 29 08:10:35.594801 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:35.594767 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" podUID="c18a6983-3874-4a51-8237-356b8f6cd8d8" containerName="payload-pre-processing" containerID="cri-o://19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724" gracePeriod=30 Jul 29 08:10:35.879548 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:35.879522 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:10:35.881353 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:35.881337 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:10:36.023386 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.023351 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqwh\" (UniqueName: \"kubernetes.io/projected/c18a6983-3874-4a51-8237-356b8f6cd8d8-kube-api-access-bgqwh\") pod \"c18a6983-3874-4a51-8237-356b8f6cd8d8\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " Jul 29 08:10:36.023582 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.023410 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6txc\" (UniqueName: \"kubernetes.io/projected/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-kube-api-access-d6txc\") pod \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " Jul 29 08:10:36.023582 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.023439 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-plugins-config-volume\") pod \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\" (UID: \"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a\") " Jul 29 08:10:36.023582 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.023512 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18a6983-3874-4a51-8237-356b8f6cd8d8-plugins-config-volume\") pod \"c18a6983-3874-4a51-8237-356b8f6cd8d8\" (UID: \"c18a6983-3874-4a51-8237-356b8f6cd8d8\") " Jul 29 08:10:36.023875 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.023848 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" (UID: "19bafdb2-154c-4f2a-ab49-ee26eb9ab80a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:36.023981 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.023899 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18a6983-3874-4a51-8237-356b8f6cd8d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c18a6983-3874-4a51-8237-356b8f6cd8d8" (UID: "c18a6983-3874-4a51-8237-356b8f6cd8d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:36.025457 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.025436 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-kube-api-access-d6txc" (OuterVolumeSpecName: "kube-api-access-d6txc") pod "19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" (UID: "19bafdb2-154c-4f2a-ab49-ee26eb9ab80a"). InnerVolumeSpecName "kube-api-access-d6txc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:36.025457 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.025446 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18a6983-3874-4a51-8237-356b8f6cd8d8-kube-api-access-bgqwh" (OuterVolumeSpecName: "kube-api-access-bgqwh") pod "c18a6983-3874-4a51-8237-356b8f6cd8d8" (UID: "c18a6983-3874-4a51-8237-356b8f6cd8d8"). InnerVolumeSpecName "kube-api-access-bgqwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:36.125194 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.125119 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6txc\" (UniqueName: \"kubernetes.io/projected/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-kube-api-access-d6txc\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:36.125194 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.125145 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:36.125194 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.125158 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c18a6983-3874-4a51-8237-356b8f6cd8d8-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:36.125194 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.125170 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqwh\" (UniqueName: \"kubernetes.io/projected/c18a6983-3874-4a51-8237-356b8f6cd8d8-kube-api-access-bgqwh\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:36.312039 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.312008 2565 generic.go:356] "Generic (PLEG): container finished" podID="19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" containerID="5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6" exitCode=0 Jul 29 08:10:36.312198 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.312047 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" Jul 29 08:10:36.312198 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.312088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" event={"ID":"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a","Type":"ContainerDied","Data":"5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6"} Jul 29 08:10:36.312198 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.312137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh" event={"ID":"19bafdb2-154c-4f2a-ab49-ee26eb9ab80a","Type":"ContainerDied","Data":"db8a56c2ebad7085e42708264f0bd61835c7ab12eb6c97409611c739429ae4c0"} Jul 29 08:10:36.312198 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.312160 2565 scope.go:117] "RemoveContainer" containerID="5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6" Jul 29 08:10:36.313491 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.313470 2565 generic.go:356] "Generic (PLEG): container finished" podID="c18a6983-3874-4a51-8237-356b8f6cd8d8" containerID="19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724" exitCode=0 Jul 29 08:10:36.313577 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.313508 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" Jul 29 08:10:36.313577 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.313536 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" event={"ID":"c18a6983-3874-4a51-8237-356b8f6cd8d8","Type":"ContainerDied","Data":"19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724"} Jul 29 08:10:36.313662 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.313576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c" event={"ID":"c18a6983-3874-4a51-8237-356b8f6cd8d8","Type":"ContainerDied","Data":"bd50dad29f2cac8453e8eed7f430020ea412b71d8c48a3cdd7dc6eb285d828a5"} Jul 29 08:10:36.321755 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.321737 2565 scope.go:117] "RemoveContainer" containerID="5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6" Jul 29 08:10:36.322147 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:10:36.322120 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6\": container with ID starting with 5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6 not found: ID does not exist" containerID="5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6" Jul 29 08:10:36.322243 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.322157 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6"} err="failed to get container status \"5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6\": rpc error: code = NotFound desc = could not find container \"5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6\": container with ID starting with 5b201b9da173cd5e5daf63b545e88335162d6c1591c2eb01fab43c3b859e8ab6 not found: ID does not exist" Jul 29 08:10:36.322243 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.322177 2565 scope.go:117] "RemoveContainer" containerID="19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724" Jul 29 08:10:36.330409 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.330392 2565 scope.go:117] "RemoveContainer" containerID="19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724" Jul 29 08:10:36.330630 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:10:36.330610 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724\": container with ID starting with 19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724 not found: ID does not exist" containerID="19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724" Jul 29 08:10:36.330701 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.330641 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724"} err="failed to get container status \"19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724\": rpc error: code = NotFound desc = could not find container \"19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724\": container with ID starting with 19a45d94e88c94706a0981536b866c2e9732fc46ca2a7e073095dc71019c2724 not found: ID does not exist" Jul 29 08:10:36.364401 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.364377 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c"] Jul 29 08:10:36.373300 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.373279 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1dc98-66544b996d-ggh2c"] Jul 29 08:10:36.401724 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.401670 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh"] Jul 29 08:10:36.408447 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:36.408427 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1dc98-7fbdbcd7f4-8rcxh"] Jul 29 08:10:37.020376 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:37.020342 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" path="/var/lib/kubelet/pods/19bafdb2-154c-4f2a-ab49-ee26eb9ab80a/volumes" Jul 29 08:10:37.020728 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:37.020722 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c18a6983-3874-4a51-8237-356b8f6cd8d8" path="/var/lib/kubelet/pods/c18a6983-3874-4a51-8237-356b8f6cd8d8/volumes" Jul 29 08:10:48.790830 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:48.790792 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs"] Jul 29 08:10:48.791278 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:48.791052 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" podUID="b31c08e8-d431-4131-9467-9cdaa8dc53cd" containerName="maas-api" containerID="cri-o://e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a" gracePeriod=30 Jul 29 08:10:48.857227 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:48.857197 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn"] Jul 29 08:10:48.857431 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:48.857393 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" podUID="fd383e36-3f7c-4ecb-9032-7b752633533f" containerName="payload-processing" containerID="cri-o://983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462" gracePeriod=30 Jul 29 08:10:48.865904 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:48.865873 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw"] Jul 29 08:10:48.866125 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:48.866099 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" podUID="2d572a38-158e-4370-973a-e635f191f0f6" containerName="payload-pre-processing" containerID="cri-o://1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e" gracePeriod=30 Jul 29 08:10:49.139818 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.139785 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 08:10:49.170917 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.170892 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 08:10:49.181716 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.181694 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 08:10:49.241003 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.240947 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzp46\" (UniqueName: \"kubernetes.io/projected/b31c08e8-d431-4131-9467-9cdaa8dc53cd-kube-api-access-jzp46\") pod \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " Jul 29 08:10:49.241143 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.241029 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls\") pod \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " Jul 29 08:10:49.241143 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.241067 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c08e8-d431-4131-9467-9cdaa8dc53cd-openshift-service-ca\") pod \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\" (UID: \"b31c08e8-d431-4131-9467-9cdaa8dc53cd\") " Jul 29 08:10:49.241450 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.241421 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31c08e8-d431-4131-9467-9cdaa8dc53cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b31c08e8-d431-4131-9467-9cdaa8dc53cd" (UID: "b31c08e8-d431-4131-9467-9cdaa8dc53cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:49.242985 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.242939 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31c08e8-d431-4131-9467-9cdaa8dc53cd-kube-api-access-jzp46" (OuterVolumeSpecName: "kube-api-access-jzp46") pod "b31c08e8-d431-4131-9467-9cdaa8dc53cd" (UID: "b31c08e8-d431-4131-9467-9cdaa8dc53cd"). InnerVolumeSpecName "kube-api-access-jzp46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:49.242985 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.242980 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b31c08e8-d431-4131-9467-9cdaa8dc53cd" (UID: "b31c08e8-d431-4131-9467-9cdaa8dc53cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:10:49.341937 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.341857 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd383e36-3f7c-4ecb-9032-7b752633533f-plugins-config-volume\") pod \"fd383e36-3f7c-4ecb-9032-7b752633533f\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " Jul 29 08:10:49.341937 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.341907 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/2d572a38-158e-4370-973a-e635f191f0f6-kube-api-access-pdsl7\") pod \"2d572a38-158e-4370-973a-e635f191f0f6\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " Jul 29 08:10:49.341937 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.341931 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d572a38-158e-4370-973a-e635f191f0f6-plugins-config-volume\") pod \"2d572a38-158e-4370-973a-e635f191f0f6\" (UID: \"2d572a38-158e-4370-973a-e635f191f0f6\") " Jul 29 08:10:49.342243 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.342016 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86c7q\" (UniqueName: \"kubernetes.io/projected/fd383e36-3f7c-4ecb-9032-7b752633533f-kube-api-access-86c7q\") pod \"fd383e36-3f7c-4ecb-9032-7b752633533f\" (UID: \"fd383e36-3f7c-4ecb-9032-7b752633533f\") " Jul 29 08:10:49.342319 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.342291 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd383e36-3f7c-4ecb-9032-7b752633533f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd383e36-3f7c-4ecb-9032-7b752633533f" (UID: "fd383e36-3f7c-4ecb-9032-7b752633533f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:49.342364 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.342345 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d572a38-158e-4370-973a-e635f191f0f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d572a38-158e-4370-973a-e635f191f0f6" (UID: "2d572a38-158e-4370-973a-e635f191f0f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:10:49.342402 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.342367 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzp46\" (UniqueName: \"kubernetes.io/projected/b31c08e8-d431-4131-9467-9cdaa8dc53cd-kube-api-access-jzp46\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.342402 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.342384 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31c08e8-d431-4131-9467-9cdaa8dc53cd-maas-api-tls\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.342402 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.342393 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c08e8-d431-4131-9467-9cdaa8dc53cd-openshift-service-ca\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.343963 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.343928 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d572a38-158e-4370-973a-e635f191f0f6-kube-api-access-pdsl7" (OuterVolumeSpecName: "kube-api-access-pdsl7") pod "2d572a38-158e-4370-973a-e635f191f0f6" (UID: "2d572a38-158e-4370-973a-e635f191f0f6"). InnerVolumeSpecName "kube-api-access-pdsl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:49.344036 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.343971 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd383e36-3f7c-4ecb-9032-7b752633533f-kube-api-access-86c7q" (OuterVolumeSpecName: "kube-api-access-86c7q") pod "fd383e36-3f7c-4ecb-9032-7b752633533f" (UID: "fd383e36-3f7c-4ecb-9032-7b752633533f"). InnerVolumeSpecName "kube-api-access-86c7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:10:49.386897 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.386866 2565 generic.go:356] "Generic (PLEG): container finished" podID="2d572a38-158e-4370-973a-e635f191f0f6" containerID="1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e" exitCode=0 Jul 29 08:10:49.387068 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.386910 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" Jul 29 08:10:49.387068 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.386969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" event={"ID":"2d572a38-158e-4370-973a-e635f191f0f6","Type":"ContainerDied","Data":"1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e"} Jul 29 08:10:49.387068 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.387029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw" event={"ID":"2d572a38-158e-4370-973a-e635f191f0f6","Type":"ContainerDied","Data":"e3a6c62645fa778e39271916d6e25bbe6143b86c4f9830e0aad2c47f0ce3fc78"} Jul 29 08:10:49.387068 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.387059 2565 scope.go:117] "RemoveContainer" containerID="1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e" Jul 29 08:10:49.388465 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.388442 2565 generic.go:356] "Generic (PLEG): container finished" podID="b31c08e8-d431-4131-9467-9cdaa8dc53cd" containerID="e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a" exitCode=0 Jul 29 08:10:49.388584 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.388484 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" Jul 29 08:10:49.388584 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.388496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" event={"ID":"b31c08e8-d431-4131-9467-9cdaa8dc53cd","Type":"ContainerDied","Data":"e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a"} Jul 29 08:10:49.388584 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.388559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs" event={"ID":"b31c08e8-d431-4131-9467-9cdaa8dc53cd","Type":"ContainerDied","Data":"c546a2713ad4f1e90b8b3331468ec8e0e463ec741099200b1168660bb1b2ec32"} Jul 29 08:10:49.390089 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.390064 2565 generic.go:356] "Generic (PLEG): container finished" podID="fd383e36-3f7c-4ecb-9032-7b752633533f" containerID="983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462" exitCode=0 Jul 29 08:10:49.390207 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.390111 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" Jul 29 08:10:49.390207 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.390110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" event={"ID":"fd383e36-3f7c-4ecb-9032-7b752633533f","Type":"ContainerDied","Data":"983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462"} Jul 29 08:10:49.390207 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.390167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn" event={"ID":"fd383e36-3f7c-4ecb-9032-7b752633533f","Type":"ContainerDied","Data":"f37c9c309399f5f49f6aa00b5e71b3debdb20d4ee534e9bce46467d2f99c489b"} Jul 29 08:10:49.398614 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.398590 2565 scope.go:117] "RemoveContainer" containerID="1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e" Jul 29 08:10:49.398904 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:10:49.398881 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e\": container with ID starting with 1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e not found: ID does not exist" containerID="1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e" Jul 29 08:10:49.398986 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.398914 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e"} err="failed to get container status \"1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e\": rpc error: code = NotFound desc = could not find container \"1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e\": container with ID starting with 1f8abe5d0847a20c1890a66b056ded97a0c9189ec2a12f9ff124d5c3a22ddc2e not found: ID does not exist" Jul 29 08:10:49.398986 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.398931 2565 scope.go:117] "RemoveContainer" containerID="e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a" Jul 29 08:10:49.406964 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.406939 2565 scope.go:117] "RemoveContainer" containerID="e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a" Jul 29 08:10:49.407216 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:10:49.407200 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a\": container with ID starting with e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a not found: ID does not exist" containerID="e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a" Jul 29 08:10:49.407294 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.407225 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a"} err="failed to get container status \"e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a\": rpc error: code = NotFound desc = could not find container \"e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a\": container with ID starting with e06d5b185ee27ced2ef847785c6e39397f78992ad9a214539cf35269bf47170a not found: ID does not exist" Jul 29 08:10:49.407294 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.407245 2565 scope.go:117] "RemoveContainer" containerID="983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462" Jul 29 08:10:49.414827 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.414811 2565 scope.go:117] "RemoveContainer" containerID="983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462" Jul 29 08:10:49.415127 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:10:49.415107 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462\": container with ID starting with 983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462 not found: ID does not exist" containerID="983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462" Jul 29 08:10:49.415188 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.415132 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462"} err="failed to get container status \"983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462\": rpc error: code = NotFound desc = could not find container \"983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462\": container with ID starting with 983744d20b5d136be2999911df306e6dbcba82edd82dee76f0044327ade55462 not found: ID does not exist" Jul 29 08:10:49.425723 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.425699 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs"] Jul 29 08:10:49.429182 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.429162 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-984886-869ddb5bbd-bgrvs"] Jul 29 08:10:49.441369 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.441348 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw"] Jul 29 08:10:49.443226 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.443203 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd383e36-3f7c-4ecb-9032-7b752633533f-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.443334 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.443231 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdsl7\" (UniqueName: \"kubernetes.io/projected/2d572a38-158e-4370-973a-e635f191f0f6-kube-api-access-pdsl7\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.443334 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.443248 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d572a38-158e-4370-973a-e635f191f0f6-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.443334 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.443263 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86c7q\" (UniqueName: \"kubernetes.io/projected/fd383e36-3f7c-4ecb-9032-7b752633533f-kube-api-access-86c7q\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:10:49.445712 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.445693 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-984886-78d44c8588-dl4fw"] Jul 29 08:10:49.456221 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.456204 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn"] Jul 29 08:10:49.460521 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:49.460502 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-984886-c56f8b66b-gkxfn"] Jul 29 08:10:51.021138 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:51.021104 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d572a38-158e-4370-973a-e635f191f0f6" path="/var/lib/kubelet/pods/2d572a38-158e-4370-973a-e635f191f0f6/volumes" Jul 29 08:10:51.021593 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:51.021481 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31c08e8-d431-4131-9467-9cdaa8dc53cd" path="/var/lib/kubelet/pods/b31c08e8-d431-4131-9467-9cdaa8dc53cd/volumes" Jul 29 08:10:51.021792 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:51.021779 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd383e36-3f7c-4ecb-9032-7b752633533f" path="/var/lib/kubelet/pods/fd383e36-3f7c-4ecb-9032-7b752633533f/volumes" Jul 29 08:10:55.998218 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:55.998186 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5"] Jul 29 08:10:55.998672 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:55.998470 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" containerID="cri-o://fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507" gracePeriod=30 Jul 29 08:10:58.591363 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:58.591329 2565 patch_prober.go:28] interesting pod/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 08:10:58.591743 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:10:58.591386 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 08:11:01.248675 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.248651 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 08:11:01.346025 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.345996 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-credential-socket\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346172 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346076 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-data\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346172 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346104 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-podinfo\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346172 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346126 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istiod-ca-cert\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346172 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346151 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5tzj\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-kube-api-access-f5tzj\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346363 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346184 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-socket\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346363 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346276 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:01.346363 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-token\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346524 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346388 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-certs\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346524 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346443 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:01.346524 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346452 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-envoy\") pod \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\" (UID: \"afd1a8cc-6319-4b51-9bf1-07905a5131b9\") " Jul 29 08:11:01.346681 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346622 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:11:01.346681 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346645 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:01.346799 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346777 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istiod-ca-cert\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.346799 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346795 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-socket\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.346908 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346810 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-workload-certs\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.346908 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346823 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-credential-socket\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.347018 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.346976 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:01.348372 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.348342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:01.348372 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.348363 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-kube-api-access-f5tzj" (OuterVolumeSpecName: "kube-api-access-f5tzj") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "kube-api-access-f5tzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:01.348614 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.348600 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 08:11:01.348737 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.348714 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afd1a8cc-6319-4b51-9bf1-07905a5131b9" (UID: "afd1a8cc-6319-4b51-9bf1-07905a5131b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:11:01.447414 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.447388 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-data\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.447414 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.447410 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-podinfo\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.447565 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.447420 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5tzj\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-kube-api-access-f5tzj\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.447565 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.447429 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-token\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.447565 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.447437 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd1a8cc-6319-4b51-9bf1-07905a5131b9-istio-envoy\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:01.460992 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.460944 2565 generic.go:356] "Generic (PLEG): container finished" podID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerID="fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507" exitCode=0 Jul 29 08:11:01.461111 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.461024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" event={"ID":"afd1a8cc-6319-4b51-9bf1-07905a5131b9","Type":"ContainerDied","Data":"fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507"} Jul 29 08:11:01.461111 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.461039 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" Jul 29 08:11:01.461111 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.461091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5" event={"ID":"afd1a8cc-6319-4b51-9bf1-07905a5131b9","Type":"ContainerDied","Data":"983a55efb445d9ea5049384a0a4f66ac8b68fa4474a4a83ec1b88423aa38c53a"} Jul 29 08:11:01.461279 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.461122 2565 scope.go:117] "RemoveContainer" containerID="fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507" Jul 29 08:11:01.470329 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.470315 2565 scope.go:117] "RemoveContainer" containerID="05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b" Jul 29 08:11:01.478113 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.478063 2565 scope.go:117] "RemoveContainer" containerID="fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507" Jul 29 08:11:01.478312 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:11:01.478292 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507\": container with ID starting with fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507 not found: ID does not exist" containerID="fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507" Jul 29 08:11:01.478352 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.478320 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507"} err="failed to get container status \"fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507\": rpc error: code = NotFound desc = could not find container \"fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507\": container with ID starting with fe4f1db974577f31915aacb76f231db6d212aac6f105d61babe7000dc22bf507 not found: ID does not exist" Jul 29 08:11:01.478352 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.478336 2565 scope.go:117] "RemoveContainer" containerID="05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b" Jul 29 08:11:01.478567 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:11:01.478553 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b\": container with ID starting with 05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b not found: ID does not exist" containerID="05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b" Jul 29 08:11:01.478602 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.478571 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b"} err="failed to get container status \"05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b\": rpc error: code = NotFound desc = could not find container \"05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b\": container with ID starting with 05be458ccf11f5a124b22ca7fc4564d5e7d880edb43cd707cf8aec209336e71b not found: ID does not exist" Jul 29 08:11:01.491318 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.491299 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5"] Jul 29 08:11:01.495601 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:01.495575 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-984886-openshift-default-6bbc5cdc44-z4jd5"] Jul 29 08:11:02.028053 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.028021 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv"] Jul 29 08:11:02.028295 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.028249 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" podUID="99567592-6525-4c92-b72c-7150562c79ed" containerName="payload-processing" containerID="cri-o://852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82" gracePeriod=30 Jul 29 08:11:02.042643 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.042613 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7"] Jul 29 08:11:02.042854 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.042832 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" podUID="d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" containerName="payload-pre-processing" containerID="cri-o://b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e" gracePeriod=30 Jul 29 08:11:02.306081 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.306059 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 08:11:02.308744 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.308726 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 08:11:02.354764 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.354737 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99567592-6525-4c92-b72c-7150562c79ed-plugins-config-volume\") pod \"99567592-6525-4c92-b72c-7150562c79ed\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " Jul 29 08:11:02.354919 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.354789 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fgnn\" (UniqueName: \"kubernetes.io/projected/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-kube-api-access-5fgnn\") pod \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " Jul 29 08:11:02.354919 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.354869 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-plugins-config-volume\") pod \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\" (UID: \"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a\") " Jul 29 08:11:02.354919 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.354911 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd4hl\" (UniqueName: \"kubernetes.io/projected/99567592-6525-4c92-b72c-7150562c79ed-kube-api-access-qd4hl\") pod \"99567592-6525-4c92-b72c-7150562c79ed\" (UID: \"99567592-6525-4c92-b72c-7150562c79ed\") " Jul 29 08:11:02.355279 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.355238 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99567592-6525-4c92-b72c-7150562c79ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99567592-6525-4c92-b72c-7150562c79ed" (UID: "99567592-6525-4c92-b72c-7150562c79ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:11:02.355422 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.355281 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" (UID: "d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:11:02.357113 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.357089 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-kube-api-access-5fgnn" (OuterVolumeSpecName: "kube-api-access-5fgnn") pod "d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" (UID: "d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a"). InnerVolumeSpecName "kube-api-access-5fgnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:02.357113 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.357101 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99567592-6525-4c92-b72c-7150562c79ed-kube-api-access-qd4hl" (OuterVolumeSpecName: "kube-api-access-qd4hl") pod "99567592-6525-4c92-b72c-7150562c79ed" (UID: "99567592-6525-4c92-b72c-7150562c79ed"). InnerVolumeSpecName "kube-api-access-qd4hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:11:02.456210 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.456176 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgnn\" (UniqueName: \"kubernetes.io/projected/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-kube-api-access-5fgnn\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.456210 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.456202 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.456210 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.456212 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd4hl\" (UniqueName: \"kubernetes.io/projected/99567592-6525-4c92-b72c-7150562c79ed-kube-api-access-qd4hl\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.456442 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.456220 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99567592-6525-4c92-b72c-7150562c79ed-plugins-config-volume\") on node \"ip-10-0-143-226.ec2.internal\" DevicePath \"\"" Jul 29 08:11:02.468436 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.468407 2565 generic.go:356] "Generic (PLEG): container finished" podID="99567592-6525-4c92-b72c-7150562c79ed" containerID="852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82" exitCode=0 Jul 29 08:11:02.468602 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.468463 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" Jul 29 08:11:02.468602 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.468508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" event={"ID":"99567592-6525-4c92-b72c-7150562c79ed","Type":"ContainerDied","Data":"852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82"} Jul 29 08:11:02.468602 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.468566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv" event={"ID":"99567592-6525-4c92-b72c-7150562c79ed","Type":"ContainerDied","Data":"33aca1d5ff78b4cc1b8ff34262156d4e585d6c296f05c4f35cfaaefca44da856"} Jul 29 08:11:02.468602 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.468600 2565 scope.go:117] "RemoveContainer" containerID="852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82" Jul 29 08:11:02.470877 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.470855 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" containerID="b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e" exitCode=0 Jul 29 08:11:02.471051 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.470887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" event={"ID":"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a","Type":"ContainerDied","Data":"b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e"} Jul 29 08:11:02.471051 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.470937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" event={"ID":"d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a","Type":"ContainerDied","Data":"44643049f33b2618580f130a4ea7ee5dbca380ef4bc093e3af5f688ea2c59f2a"} Jul 29 08:11:02.471051 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.470899 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7" Jul 29 08:11:02.477902 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.477879 2565 scope.go:117] "RemoveContainer" containerID="852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82" Jul 29 08:11:02.478221 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:11:02.478186 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82\": container with ID starting with 852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82 not found: ID does not exist" containerID="852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82" Jul 29 08:11:02.478317 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.478221 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82"} err="failed to get container status \"852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82\": rpc error: code = NotFound desc = could not find container \"852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82\": container with ID starting with 852984c2e993cb1c0257f240f5e6ccefcd105536c01a351ea4fc47b0467e9c82 not found: ID does not exist" Jul 29 08:11:02.478317 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.478244 2565 scope.go:117] "RemoveContainer" containerID="b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e" Jul 29 08:11:02.486235 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.486218 2565 scope.go:117] "RemoveContainer" containerID="b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e" Jul 29 08:11:02.486490 ip-10-0-143-226 kubenswrapper[2565]: E0729 08:11:02.486472 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e\": container with ID starting with b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e not found: ID does not exist" containerID="b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e" Jul 29 08:11:02.486542 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.486497 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e"} err="failed to get container status \"b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e\": rpc error: code = NotFound desc = could not find container \"b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e\": container with ID starting with b64f1c85119054eba068b46d6a82889a75eaaab7cd3b67400998fa4b243cc69e not found: ID does not exist" Jul 29 08:11:02.493598 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.493576 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv"] Jul 29 08:11:02.497352 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.497333 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-71e038-5c994dd8c4-rtqvv"] Jul 29 08:11:02.508890 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.508869 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7"] Jul 29 08:11:02.513406 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:02.513386 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-71e038-94f457c68-t92r7"] Jul 29 08:11:03.019345 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:03.019315 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99567592-6525-4c92-b72c-7150562c79ed" path="/var/lib/kubelet/pods/99567592-6525-4c92-b72c-7150562c79ed/volumes" Jul 29 08:11:03.019735 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:03.019720 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" path="/var/lib/kubelet/pods/afd1a8cc-6319-4b51-9bf1-07905a5131b9/volumes" Jul 29 08:11:03.020085 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:03.020072 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" path="/var/lib/kubelet/pods/d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a/volumes" Jul 29 08:11:17.608222 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:17.608194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-9c4ps_a8e2f522-b621-4c5f-b340-8d81eea2dc37/copy-manifests/0.log" Jul 29 08:11:17.625085 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:17.625048 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-9c4ps_a8e2f522-b621-4c5f-b340-8d81eea2dc37/manager/0.log" Jul 29 08:11:19.611057 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.611022 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r_35ece30b-f78d-4175-83df-e665307b551c/extract/0.log" Jul 29 08:11:19.616488 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.616469 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r_35ece30b-f78d-4175-83df-e665307b551c/util/0.log" Jul 29 08:11:19.622514 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.622496 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r_35ece30b-f78d-4175-83df-e665307b551c/pull/0.log" Jul 29 08:11:19.737266 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.737217 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb_9d2d8b8b-30db-472a-9d44-2b9b54d44320/util/0.log" Jul 29 08:11:19.742661 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.742640 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb_9d2d8b8b-30db-472a-9d44-2b9b54d44320/pull/0.log" Jul 29 08:11:19.748003 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.747984 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb_9d2d8b8b-30db-472a-9d44-2b9b54d44320/extract/0.log" Jul 29 08:11:19.863296 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.863271 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb_c496b17e-3fe8-4094-916f-4ddfb88812f9/util/0.log" Jul 29 08:11:19.870531 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.870513 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb_c496b17e-3fe8-4094-916f-4ddfb88812f9/pull/0.log" Jul 29 08:11:19.877671 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.877650 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb_c496b17e-3fe8-4094-916f-4ddfb88812f9/extract/0.log" Jul 29 08:11:19.999370 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:19.999311 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2_aa36ba9a-0a57-4151-9e05-891e03cd76eb/util/0.log" Jul 29 08:11:20.020461 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:20.020407 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2_aa36ba9a-0a57-4151-9e05-891e03cd76eb/pull/0.log" Jul 29 08:11:20.027134 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:20.027102 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2_aa36ba9a-0a57-4151-9e05-891e03cd76eb/extract/0.log" Jul 29 08:11:21.427495 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:21.427465 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps_65f32fe0-45b7-4605-bf8b-43370da88b1c/istio-proxy/0.log" Jul 29 08:11:21.547832 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:21.547804 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jm2kl_142741f2-587b-4bd3-8dd5-a514b4ea172b/discovery/0.log" Jul 29 08:11:21.913290 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:21.913250 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/1.log" Jul 29 08:11:22.044606 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:22.044576 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-q6wfb_d2875f89-c8e0-4337-aee1-f043919045df/payload-pre-processing/0.log" Jul 29 08:11:22.177499 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:22.177418 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8npf2_855fe327-9db6-4515-a618-2ee91f801bc8/payload-processing/0.log" Jul 29 08:11:22.683515 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:22.683486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-2stzw_16aa9a41-1ca6-4798-b278-577bf7249aed/main/0.log" Jul 29 08:11:23.044862 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:23.044785 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5pw2z_d49480a1-37bb-49cb-a25d-260f0933d697/main/0.log" Jul 29 08:11:32.109964 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:32.109925 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vhxbk_f31a0c1a-5717-41ba-a99f-b218218bbbf6/global-pull-secret-syncer/0.log" Jul 29 08:11:32.279234 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:32.279204 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sgp6l_342f3143-0c98-4b3b-b48d-a95ba0044fd7/konnectivity-agent/0.log" Jul 29 08:11:32.369562 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:32.369498 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-226.ec2.internal_e309e9e9463c00491d4bf68a40b67053/haproxy/0.log" Jul 29 08:11:36.945653 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:36.945622 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r_35ece30b-f78d-4175-83df-e665307b551c/extract/0.log" Jul 29 08:11:36.990254 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:36.990226 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r_35ece30b-f78d-4175-83df-e665307b551c/util/0.log" Jul 29 08:11:37.069418 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.069394 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hfj6r_35ece30b-f78d-4175-83df-e665307b551c/pull/0.log" Jul 29 08:11:37.143671 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.143650 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb_9d2d8b8b-30db-472a-9d44-2b9b54d44320/extract/0.log" Jul 29 08:11:37.183556 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.183533 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb_9d2d8b8b-30db-472a-9d44-2b9b54d44320/util/0.log" Jul 29 08:11:37.233612 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.233558 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lt6gb_9d2d8b8b-30db-472a-9d44-2b9b54d44320/pull/0.log" Jul 29 08:11:37.293652 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.293615 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb_c496b17e-3fe8-4094-916f-4ddfb88812f9/extract/0.log" Jul 29 08:11:37.331497 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.331479 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb_c496b17e-3fe8-4094-916f-4ddfb88812f9/util/0.log" Jul 29 08:11:37.370217 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.370197 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lqjrb_c496b17e-3fe8-4094-916f-4ddfb88812f9/pull/0.log" Jul 29 08:11:37.416432 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.416411 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2_aa36ba9a-0a57-4151-9e05-891e03cd76eb/extract/0.log" Jul 29 08:11:37.455987 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.455967 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2_aa36ba9a-0a57-4151-9e05-891e03cd76eb/util/0.log" Jul 29 08:11:37.490214 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:37.490134 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b87d2_aa36ba9a-0a57-4151-9e05-891e03cd76eb/pull/0.log" Jul 29 08:11:40.353546 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.353518 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/alertmanager/0.log" Jul 29 08:11:40.399041 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.399017 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/config-reloader/0.log" Jul 29 08:11:40.423508 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.423485 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/kube-rbac-proxy-web/0.log" Jul 29 08:11:40.447032 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.447013 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/kube-rbac-proxy/0.log" Jul 29 08:11:40.468530 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.468511 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/kube-rbac-proxy-metric/0.log" Jul 29 08:11:40.492591 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.492571 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/prom-label-proxy/0.log" Jul 29 08:11:40.517775 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.517757 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbfab147-2e2f-42fb-a0e9-2d684911c1b4/init-config-reloader/0.log" Jul 29 08:11:40.598171 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.598147 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-stcxd_459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0/kube-state-metrics/0.log" Jul 29 08:11:40.623829 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.623807 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-stcxd_459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0/kube-rbac-proxy-main/0.log" Jul 29 08:11:40.648650 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.648632 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-stcxd_459fc0f5-ba7b-4b9c-aa6a-4687ea13f8e0/kube-rbac-proxy-self/0.log" Jul 29 08:11:40.757194 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.757154 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5br72_7177cd99-fdf1-49b1-9ca2-70739ef9c4de/node-exporter/0.log" Jul 29 08:11:40.784616 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.784596 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5br72_7177cd99-fdf1-49b1-9ca2-70739ef9c4de/kube-rbac-proxy/0.log" Jul 29 08:11:40.809650 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:40.809633 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5br72_7177cd99-fdf1-49b1-9ca2-70739ef9c4de/init-textfile/0.log" Jul 29 08:11:41.363891 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.363863 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56d4bf98f7-fw9z7_b2453c3d-97fb-45f2-8062-cb27f05b8717/telemeter-client/0.log" Jul 29 08:11:41.385862 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.385838 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56d4bf98f7-fw9z7_b2453c3d-97fb-45f2-8062-cb27f05b8717/reload/0.log" Jul 29 08:11:41.407722 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.407701 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56d4bf98f7-fw9z7_b2453c3d-97fb-45f2-8062-cb27f05b8717/kube-rbac-proxy/0.log" Jul 29 08:11:41.435602 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.435576 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6898c4698c-gl8g2_19e3cb62-307a-40af-b497-3afb406dc032/thanos-query/0.log" Jul 29 08:11:41.457479 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.457460 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6898c4698c-gl8g2_19e3cb62-307a-40af-b497-3afb406dc032/kube-rbac-proxy-web/0.log" Jul 29 08:11:41.479336 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.479317 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6898c4698c-gl8g2_19e3cb62-307a-40af-b497-3afb406dc032/kube-rbac-proxy/0.log" Jul 29 08:11:41.505429 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.505406 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6898c4698c-gl8g2_19e3cb62-307a-40af-b497-3afb406dc032/prom-label-proxy/0.log" Jul 29 08:11:41.529143 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.529120 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6898c4698c-gl8g2_19e3cb62-307a-40af-b497-3afb406dc032/kube-rbac-proxy-rules/0.log" Jul 29 08:11:41.552890 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:41.552874 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6898c4698c-gl8g2_19e3cb62-307a-40af-b497-3afb406dc032/kube-rbac-proxy-metrics/0.log" Jul 29 08:11:43.212831 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.212798 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c"] Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213149 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d572a38-158e-4370-973a-e635f191f0f6" containerName="payload-pre-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213161 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d572a38-158e-4370-973a-e635f191f0f6" containerName="payload-pre-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213171 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31c08e8-d431-4131-9467-9cdaa8dc53cd" containerName="maas-api" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213177 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31c08e8-d431-4131-9467-9cdaa8dc53cd" containerName="maas-api" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213187 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" containerName="payload-pre-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213193 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" containerName="payload-pre-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213202 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99567592-6525-4c92-b72c-7150562c79ed" containerName="payload-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213208 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99567592-6525-4c92-b72c-7150562c79ed" containerName="payload-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213215 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213220 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213228 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" containerName="payload-processing" Jul 29 08:11:43.213231 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213233 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213240 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213245 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213255 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd383e36-3f7c-4ecb-9032-7b752633533f" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213261 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd383e36-3f7c-4ecb-9032-7b752633533f" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213266 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18a6983-3874-4a51-8237-356b8f6cd8d8" containerName="payload-pre-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213271 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18a6983-3874-4a51-8237-356b8f6cd8d8" containerName="payload-pre-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213279 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b8ffc4-5ab4-431a-bb4c-aad906fab837" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213283 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b8ffc4-5ab4-431a-bb4c-aad906fab837" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213293 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824ecf8b-a026-4873-925e-cf8a4fd71177" containerName="maas-api" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213298 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="824ecf8b-a026-4873-925e-cf8a4fd71177" containerName="maas-api" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213348 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bafdb2-154c-4f2a-ab49-ee26eb9ab80a" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213354 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd1a8cc-6319-4b51-9bf1-07905a5131b9" containerName="istio-proxy" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213360 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd383e36-3f7c-4ecb-9032-7b752633533f" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213366 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="824ecf8b-a026-4873-925e-cf8a4fd71177" containerName="maas-api" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213372 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d572a38-158e-4370-973a-e635f191f0f6" containerName="payload-pre-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213377 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99567592-6525-4c92-b72c-7150562c79ed" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213382 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c18a6983-3874-4a51-8237-356b8f6cd8d8" containerName="payload-pre-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213387 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d8e380-bc4b-4a1b-97d7-7fdcf0c50c8a" containerName="payload-pre-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213394 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b8ffc4-5ab4-431a-bb4c-aad906fab837" containerName="payload-processing" Jul 29 08:11:43.213608 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.213401 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31c08e8-d431-4131-9467-9cdaa8dc53cd" containerName="maas-api" Jul 29 08:11:43.216321 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.216303 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.219688 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.219667 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ztqbj\"/\"default-dockercfg-jwhlp\"" Jul 29 08:11:43.220192 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.220176 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ztqbj\"/\"openshift-service-ca.crt\"" Jul 29 08:11:43.221130 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.221113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ztqbj\"/\"kube-root-ca.crt\"" Jul 29 08:11:43.263571 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.263550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c"] Jul 29 08:11:43.296006 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.295978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-proc\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.296132 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.296012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-lib-modules\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.296200 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.296163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-sys\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.296261 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.296221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-podres\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.296261 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.296252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jrj\" (UniqueName: \"kubernetes.io/projected/bec6c75c-f150-458a-962b-417bf9bfa691-kube-api-access-49jrj\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397399 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-sys\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397416 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-podres\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jrj\" (UniqueName: \"kubernetes.io/projected/bec6c75c-f150-458a-962b-417bf9bfa691-kube-api-access-49jrj\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397519 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-podres\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-proc\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-proc\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-lib-modules\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397587 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-sys\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.397893 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.397662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bec6c75c-f150-458a-962b-417bf9bfa691-lib-modules\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.410533 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.410509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jrj\" (UniqueName: \"kubernetes.io/projected/bec6c75c-f150-458a-962b-417bf9bfa691-kube-api-access-49jrj\") pod \"perf-node-gather-daemonset-7zf5c\" (UID: \"bec6c75c-f150-458a-962b-417bf9bfa691\") " pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.525908 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.525849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:43.680236 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.680209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c"] Jul 29 08:11:43.681563 ip-10-0-143-226 kubenswrapper[2565]: W0729 08:11:43.681535 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbec6c75c_f150_458a_962b_417bf9bfa691.slice/crio-adc61ca2989a5ad1a51b8ff5e7be6b4075c7358ed440323d05529112ed918de4 WatchSource:0}: Error finding container adc61ca2989a5ad1a51b8ff5e7be6b4075c7358ed440323d05529112ed918de4: Status 404 returned error can't find the container with id adc61ca2989a5ad1a51b8ff5e7be6b4075c7358ed440323d05529112ed918de4 Jul 29 08:11:43.925196 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:43.925173 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c47498bb6-rmxdn_8eb685d3-7d87-4628-8564-a4bd967d1b7c/console/0.log" Jul 29 08:11:44.689531 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:44.689498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" event={"ID":"bec6c75c-f150-458a-962b-417bf9bfa691","Type":"ContainerStarted","Data":"31738594d68a292c427d19af5764432fd5712fab0e5ecc6142bd6d0f26be9f96"} Jul 29 08:11:44.689941 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:44.689548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" event={"ID":"bec6c75c-f150-458a-962b-417bf9bfa691","Type":"ContainerStarted","Data":"adc61ca2989a5ad1a51b8ff5e7be6b4075c7358ed440323d05529112ed918de4"} Jul 29 08:11:44.689941 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:44.689581 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:44.724519 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:44.724472 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" podStartSLOduration=1.724460384 podStartE2EDuration="1.724460384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:11:44.719930659 +0000 UTC m=+3386.326108766" watchObservedRunningTime="2026-07-29 08:11:44.724460384 +0000 UTC m=+3386.330638491" Jul 29 08:11:46.065492 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:46.065466 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4djsx_d8cb8b35-e0d0-46cf-9d36-ab496045c246/dns/0.log" Jul 29 08:11:46.089356 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:46.089333 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4djsx_d8cb8b35-e0d0-46cf-9d36-ab496045c246/kube-rbac-proxy/0.log" Jul 29 08:11:46.305987 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:46.305936 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pqrbt_b38d9af8-cc44-431b-b3a9-59d63f50990c/dns-node-resolver/0.log" Jul 29 08:11:46.974971 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:46.974917 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7fd6c64cd-c89jh_cbea23ef-06fe-448e-b3da-793ce401fac1/registry/0.log" Jul 29 08:11:47.232751 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:47.232683 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r4v44_9d7c6a4b-d721-4564-9325-0c0aaf987eed/node-ca/0.log" Jul 29 08:11:48.471193 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:48.471149 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps_65f32fe0-45b7-4605-bf8b-43370da88b1c/istio-proxy/0.log" Jul 29 08:11:48.639263 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:48.639232 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jm2kl_142741f2-587b-4bd3-8dd5-a514b4ea172b/discovery/0.log" Jul 29 08:11:48.798515 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:48.798432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/0.log" Jul 29 08:11:48.822897 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:48.822873 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/1.log" Jul 29 08:11:48.868930 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:48.868905 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-q6wfb_d2875f89-c8e0-4337-aee1-f043919045df/payload-pre-processing/0.log" Jul 29 08:11:48.907342 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:48.907320 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8npf2_855fe327-9db6-4515-a618-2ee91f801bc8/payload-processing/0.log" Jul 29 08:11:49.910408 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:49.910380 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55xw5_97771b42-fccd-49b0-9b43-f7d1869ace3a/serve-healthcheck-canary/0.log" Jul 29 08:11:50.659911 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:50.659883 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qvxl2_0faf0f45-8518-4a64-9506-e499f0d4e3fa/kube-rbac-proxy/0.log" Jul 29 08:11:50.683581 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:50.683557 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qvxl2_0faf0f45-8518-4a64-9506-e499f0d4e3fa/exporter/0.log" Jul 29 08:11:50.704172 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:50.704148 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ztqbj/perf-node-gather-daemonset-7zf5c" Jul 29 08:11:50.712377 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:50.712359 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qvxl2_0faf0f45-8518-4a64-9506-e499f0d4e3fa/extractor/0.log" Jul 29 08:11:52.679386 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:52.679351 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps_65f32fe0-45b7-4605-bf8b-43370da88b1c/istio-proxy/0.log" Jul 29 08:11:52.818983 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:52.818934 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jm2kl_142741f2-587b-4bd3-8dd5-a514b4ea172b/discovery/0.log" Jul 29 08:11:52.886283 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:52.886210 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/0.log" Jul 29 08:11:52.912914 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:52.912895 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/1.log" Jul 29 08:11:52.944652 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:52.944628 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-q6wfb_d2875f89-c8e0-4337-aee1-f043919045df/payload-pre-processing/0.log" Jul 29 08:11:52.965162 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:52.965129 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8npf2_855fe327-9db6-4515-a618-2ee91f801bc8/payload-processing/0.log" Jul 29 08:11:54.228182 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.228151 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-9c4ps_a8e2f522-b621-4c5f-b340-8d81eea2dc37/manager/0.log" Jul 29 08:11:54.319987 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.319963 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6c966467-9c4ps_a8e2f522-b621-4c5f-b340-8d81eea2dc37/copy-manifests/0.log" Jul 29 08:11:54.490853 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.490782 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-kg9ps_65f32fe0-45b7-4605-bf8b-43370da88b1c/istio-proxy/0.log" Jul 29 08:11:54.639911 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.639884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jm2kl_142741f2-587b-4bd3-8dd5-a514b4ea172b/discovery/0.log" Jul 29 08:11:54.810062 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.810036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/0.log" Jul 29 08:11:54.834670 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.834638 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vr8nm_a2fd1ef4-4586-455d-99f0-875fccfcdd72/istio-proxy/1.log" Jul 29 08:11:54.897776 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.897749 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-q6wfb_d2875f89-c8e0-4337-aee1-f043919045df/payload-pre-processing/0.log" Jul 29 08:11:54.943574 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:54.943546 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8npf2_855fe327-9db6-4515-a618-2ee91f801bc8/payload-processing/0.log" Jul 29 08:11:55.915861 ip-10-0-143-226 kubenswrapper[2565]: I0729 08:11:55.915835 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d85576cc8-79swl_dc077e91-8114-48e1-906c-ee9473f47196/manager/0.log"