Jul 18 13:32:27.992407 ip-10-0-136-156 systemd[1]: Starting Kubernetes Kubelet... Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.376748 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381431 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381451 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381459 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381473 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381479 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381484 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381488 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 13:32:28.500945 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381494 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 13:32:28.453446 ip-10-0-136-156 systemd[1]: Started Kubernetes Kubelet. Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381498 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381504 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381508 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381514 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381518 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381521 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381529 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381537 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381542 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381546 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381552 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381556 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381563 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381568 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381574 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381578 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381584 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381588 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381592 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 13:32:28.502078 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381598 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381602 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381608 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381612 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381616 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381619 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381622 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381625 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381633 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381639 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381646 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381653 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381659 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381667 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381674 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381678 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381682 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381688 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381693 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381699 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 13:32:28.503712 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381703 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381709 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381714 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381721 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381725 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381728 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381734 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381738 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381744 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381748 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381752 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381758 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381762 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381768 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381772 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381778 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381782 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381786 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381792 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381796 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381802 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 13:32:28.504451 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381808 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381813 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381818 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381822 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381825 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381827 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381829 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381836 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381840 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381846 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381850 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381856 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381860 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381864 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381870 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381874 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381880 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381884 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381887 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 13:32:28.505097 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381894 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381897 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381903 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381907 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381911 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381917 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381921 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381926 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381930 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381933 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381940 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.381943 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382955 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382966 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382973 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382976 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382980 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382985 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 13:32:28.505702 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382988 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382992 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.382997 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383000 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383005 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383008 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383010 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383013 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383018 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383021 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383023 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383027 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383030 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383032 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383035 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383037 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383040 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383043 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383049 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 13:32:28.506310 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383052 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383054 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383059 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383062 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383064 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383069 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383072 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383075 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383077 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383082 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383086 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383089 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383094 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383096 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383099 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383103 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383106 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383108 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383111 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383116 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 13:32:28.506960 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383118 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383121 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383125 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383143 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383145 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383150 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383153 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383156 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383160 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383163 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383166 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383171 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383174 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383176 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383179 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383409 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383428 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383431 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383434 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 13:32:28.507638 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383437 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383440 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383445 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383448 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383451 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383456 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383459 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383468 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383472 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383475 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383480 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383482 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383485 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383489 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383492 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383495 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383497 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383503 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383505 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383508 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 13:32:28.508255 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383515 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383518 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383520 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383525 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383528 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383530 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383535 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383538 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383541 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383544 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383548 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383551 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383554 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383561 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.383564 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384536 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384548 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384553 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384558 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384563 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 13:32:28.508891 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384567 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384572 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384576 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384580 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384585 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384596 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384600 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384604 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384612 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384619 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384623 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384628 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384632 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384637 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384641 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384645 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384649 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384654 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384664 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 13:32:28.509583 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384668 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384675 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384680 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384685 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384689 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384694 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384700 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384704 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384708 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384713 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384723 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384728 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384732 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384737 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384740 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384745 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384749 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384754 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384758 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 13:32:28.510204 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384763 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384767 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384772 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384776 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384784 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384789 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384793 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384797 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384801 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384806 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384811 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384814 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384819 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384823 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384828 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384832 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384835 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384841 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384844 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384847 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 13:32:28.510738 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384849 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.384866 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385348 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385355 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385359 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385363 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385399 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385403 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385406 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385409 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385412 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385415 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385418 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385422 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385424 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385427 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385429 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385432 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385435 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385438 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 13:32:28.511344 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385441 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385444 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385447 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385450 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385452 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385456 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385458 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385461 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385464 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385466 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385469 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385472 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385474 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385477 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385479 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.385482 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385555 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385563 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385570 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385575 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385579 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 18 13:32:28.511893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385583 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385587 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385591 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385594 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385597 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385601 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385604 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385608 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385611 2564 flags.go:64] FLAG: --cgroup-root="" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385614 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385617 2564 flags.go:64] FLAG: --client-ca-file="" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385620 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385624 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385628 2564 flags.go:64] FLAG: --cluster-domain="" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385632 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385635 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385639 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385642 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385646 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385649 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385652 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385656 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385659 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385662 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 18 13:32:28.512488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385666 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385669 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385672 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385676 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385679 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385682 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385685 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385689 2564 flags.go:64] FLAG: --enable-server="true" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385692 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385697 2564 flags.go:64] FLAG: --event-burst="100" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385700 2564 flags.go:64] FLAG: --event-qps="50" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385703 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385706 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385709 2564 flags.go:64] FLAG: --eviction-hard="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385712 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385715 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385719 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385722 2564 flags.go:64] FLAG: --eviction-soft="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385725 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385728 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385731 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385734 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385737 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385740 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385743 2564 flags.go:64] FLAG: --feature-gates="" Jul 18 13:32:28.513454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385747 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385750 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385754 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385757 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385760 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385763 2564 flags.go:64] FLAG: --help="false" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385766 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-136-156.ec2.internal" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385769 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385772 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385775 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385778 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385782 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385785 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385788 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385791 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385794 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385797 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385800 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385803 2564 flags.go:64] FLAG: --kube-reserved="" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385806 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385808 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385812 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385819 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385822 2564 flags.go:64] FLAG: --lock-file="" Jul 18 13:32:28.514494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385827 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385829 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385833 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385838 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385841 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385844 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385847 2564 flags.go:64] FLAG: --logging-format="text" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385850 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385853 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385856 2564 flags.go:64] FLAG: --manifest-url="" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385859 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385864 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385867 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385871 2564 flags.go:64] FLAG: --max-pods="110" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385874 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385877 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385880 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385883 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385886 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385889 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385892 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385899 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385902 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385905 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 18 13:32:28.515556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385908 2564 flags.go:64] FLAG: --pod-cidr="" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385911 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385917 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385920 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385923 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385926 2564 flags.go:64] FLAG: --port="10250" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385929 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385936 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0105c8d0146032715" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385940 2564 flags.go:64] FLAG: --qos-reserved="" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385943 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385946 2564 flags.go:64] FLAG: --register-node="true" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385950 2564 flags.go:64] FLAG: --register-with-taints="" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385953 2564 flags.go:64] FLAG: --registry-burst="10" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385956 2564 flags.go:64] FLAG: --registry-qps="5" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385959 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385962 2564 flags.go:64] FLAG: --reserved-memory="" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385965 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385968 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385971 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385974 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385977 2564 flags.go:64] FLAG: --runonce="false" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385980 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385983 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385986 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385989 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 18 13:32:28.516395 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385992 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385995 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.385998 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386001 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386003 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386006 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386009 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386012 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386015 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386018 2564 flags.go:64] FLAG: --system-cgroups="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386021 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386027 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386030 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386033 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386038 2564 flags.go:64] FLAG: --tls-min-version="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386043 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386046 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386049 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386052 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386055 2564 flags.go:64] FLAG: --v="2" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386064 2564 flags.go:64] FLAG: --version="false" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386069 2564 flags.go:64] FLAG: --vmodule="" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386073 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386076 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 18 13:32:28.517418 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386205 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386211 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386214 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386218 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386221 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386224 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386227 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386230 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386232 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386235 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386237 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386240 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386244 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386247 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386250 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386254 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386257 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386260 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 13:32:28.518069 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386263 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386266 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386269 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386271 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386276 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386280 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386283 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386286 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386288 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386291 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386293 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386296 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386298 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386301 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386304 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386306 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386309 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386312 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386314 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386317 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 13:32:28.518596 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386320 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386323 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386325 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386328 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386331 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386333 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386336 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386338 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386341 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386344 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386346 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386349 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386352 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386354 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386357 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386359 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386363 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386367 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386370 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386373 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 13:32:28.519489 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386376 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386378 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386381 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386383 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386386 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386388 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386391 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386393 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386396 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386398 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386401 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386403 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386406 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386408 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386411 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386413 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386416 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386419 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386422 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386424 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 13:32:28.520217 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386427 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386429 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386431 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386434 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386436 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386439 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386441 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386444 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386448 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386451 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386454 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386456 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386460 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386462 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386465 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386467 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386470 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386472 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386475 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386477 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 13:32:30.050914 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.386480 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.386934 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.394960 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.394981 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.395000 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.395005 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395063 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395069 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395072 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395075 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395078 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395081 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395084 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395087 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 13:32:30.053151 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395089 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395092 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395095 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395098 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395101 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395103 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395106 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395110 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395116 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395120 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395123 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395127 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395144 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395148 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395151 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395154 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395158 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395160 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395163 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 13:32:30.053798 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395166 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395168 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395174 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395177 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395180 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395183 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395185 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395188 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395191 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395193 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395196 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395199 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395201 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395204 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395206 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395209 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395212 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395215 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395217 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395220 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 13:32:30.054356 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395223 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395226 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395228 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395231 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395234 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395236 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395239 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395242 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395244 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395247 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395250 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395253 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395256 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395258 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395261 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395264 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395267 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395270 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395272 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395275 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 13:32:30.054937 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395277 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395280 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395282 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395285 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395287 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395290 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395293 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395295 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395298 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395300 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395303 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395306 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395309 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395311 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395314 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395317 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395319 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395322 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395324 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395327 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 13:32:30.055527 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395329 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395337 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395340 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395343 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395345 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395348 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395351 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395353 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395356 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395359 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395361 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395365 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.395371 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395487 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395491 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 13:32:30.056083 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395494 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395497 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395500 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395503 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395505 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395508 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395511 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395513 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395516 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395519 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395522 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395524 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395527 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395530 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395533 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395535 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395538 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395540 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395545 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 13:32:30.056538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395548 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395551 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395555 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395557 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395560 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395563 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395568 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395571 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395574 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395576 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395579 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395582 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395586 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395589 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395592 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395595 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395598 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395601 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395604 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 13:32:30.057087 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395607 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395609 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395612 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395615 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395618 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395620 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395623 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395626 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395628 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395631 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395633 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395636 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395639 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395642 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395644 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395647 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395649 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395652 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395654 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 13:32:30.057666 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395659 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395661 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395665 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395667 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395670 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395672 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395675 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395677 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395680 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395683 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395685 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395688 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395690 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395693 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395695 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395698 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395700 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395703 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395706 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395708 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 13:32:30.058231 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395711 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395713 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395716 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395719 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395721 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395725 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395728 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395730 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395733 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395736 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395738 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395741 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395745 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395748 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395752 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395756 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395758 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395761 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395764 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 13:32:30.058806 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:28.395767 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.395772 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.395914 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.399067 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.400244 2564 server.go:1019] "Starting client certificate rotation" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.400340 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.400410 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.402297 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.403419 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.417537 2564 log.go:25] "Validated CRI v1 runtime API" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.420680 2564 log.go:25] "Validated CRI v1 image API" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.421573 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 18 13:32:30.059339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.423740 2564 fs.go:135] Filesystem UUIDs: map[0910d3fd-25af-4308-aaa8-8c21dfe3292e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 80e72214-5fc8-4ae0-9710-5761723b2fc9:/dev/nvme0n1p4] Jul 18 13:32:30.059729 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.423757 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 18 13:32:30.059729 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.426698 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.429038 2564 manager.go:216] Machine: {Timestamp:2026-07-18 13:32:28.427744623 +0000 UTC m=+0.321552374 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102097 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25478d473504410039b871b0e5deba SystemUUID:ec25478d-4735-0441-0039-b871b0e5deba BootID:d691ec27-c1cb-4c7a-aaaf-eafde9d9df06 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:99:6d:27:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:99:6d:27:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:08:4e:bc:24:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.429154 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.429282 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.433045 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.433069 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.433751 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.433762 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.433769 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.434297 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.435347 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438259 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438400 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438432 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438445 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438459 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438468 2564 kubelet.go:398] "Adding apiserver pod source" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438479 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 18 13:32:30.059928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438858 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.438879 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.446418 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.447777 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.447806 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448346 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448362 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448368 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448374 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448384 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448392 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448403 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448412 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448422 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448430 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448441 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448452 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.448539 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rcwhr" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.449154 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.449166 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.451223 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.451222 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.452491 2564 server.go:1268] "Started kubelet" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.452582 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.452595 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.452688 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.455854 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.456204 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rcwhr" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.456545 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.461263 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18c36590734544ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:ip-10-0-136-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-07-18 13:32:28.452463818 +0000 UTC m=+0.346271566,LastTimestamp:2026-07-18 13:32:28.452463818 +0000 UTC m=+0.346271566,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.462676 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.462675 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.462851 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.463264 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.463266 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.463280 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.463383 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.463395 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465054 2564 factory.go:55] Registering systemd factory Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465069 2564 factory.go:223] Registration of the systemd container factory successfully Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465315 2564 factory.go:153] Registering CRI-O factory Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465331 2564 factory.go:223] Registration of the crio container factory successfully Jul 18 13:32:30.060524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465376 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465401 2564 factory.go:103] Registering Raw factory Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465417 2564 manager.go:1196] Started watching for new ooms in manager Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.465603 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.465822 2564 manager.go:349] Starting recovery of all containers Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.466908 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.469856 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.470829 2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-156.ec2.internal" not found Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.476304 2564 manager.go:354] Recovery completed Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.481922 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.483825 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.483852 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.483865 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.484557 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.484565 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.484568 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.484583 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485592 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485622 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485635 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485702 2564 policy_none.go:49] "None policy: Start" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485714 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485725 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.485858 2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-156.ec2.internal" not found Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.486726 2564 policy_none.go:47] "Start" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.519717 2564 manager.go:337] "Starting Device Plugin manager" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.519820 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.519834 2564 server.go:87] "Starting device plugin registration server" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.520106 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.520119 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.520249 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.520387 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.520397 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.522429 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.522473 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.542837 2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-156.ec2.internal" not found Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.620489 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.620490 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.621599 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.621643 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.621665 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.621701 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.622177 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.622209 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.622223 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.622257 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.623882 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:28.629302 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.061649 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.629321 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-156.ec2.internal\": node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.063071 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:28.652011 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.066660 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal"] Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.066741 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.068369 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.068415 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.068427 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.069790 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.069909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.069939 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.070527 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.070527 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.070588 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.070601 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.070555 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.070621 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.071677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.071700 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.072332 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.072354 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.072366 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.096534 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.107159 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.111463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e775caa402aa8a152d18579ae9d26d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"5e775caa402aa8a152d18579ae9d26d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.111495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e775caa402aa8a152d18579ae9d26d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"5e775caa402aa8a152d18579ae9d26d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.111519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d7a9cf36e12cd844e88b9ba2474ad903-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"d7a9cf36e12cd844e88b9ba2474ad903\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.111633 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.197521 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.214440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e775caa402aa8a152d18579ae9d26d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"5e775caa402aa8a152d18579ae9d26d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.232698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.214466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e775caa402aa8a152d18579ae9d26d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"5e775caa402aa8a152d18579ae9d26d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.233644 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.214483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d7a9cf36e12cd844e88b9ba2474ad903-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"d7a9cf36e12cd844e88b9ba2474ad903\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.233644 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.214523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d7a9cf36e12cd844e88b9ba2474ad903-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"d7a9cf36e12cd844e88b9ba2474ad903\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.233644 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.214541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e775caa402aa8a152d18579ae9d26d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"5e775caa402aa8a152d18579ae9d26d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.233644 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.214548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e775caa402aa8a152d18579ae9d26d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"5e775caa402aa8a152d18579ae9d26d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.300671 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.298358 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.440207 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.399672 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.440207 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.408921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.440207 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.419185 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 18 13:32:30.501422 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.500360 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.601212 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.601180 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.701746 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.701720 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.744273 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.744237 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 18 13:32:30.744427 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.744409 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 13:32:30.744467 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.744430 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 13:32:30.793076 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.793006 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 13:32:30.802326 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.802297 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.807479 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.807441 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 13:27:29 +0000 UTC" deadline="2028-01-03 06:53:03.505398709 +0000 UTC" Jul 18 13:32:30.807479 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.807476 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12809h20m32.697926051s" Jul 18 13:32:30.902996 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:30.902958 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 18 13:32:30.932160 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:30.932135 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 13:32:31.007637 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.007614 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 18 13:32:31.018008 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.017988 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 13:32:31.018947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.018925 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 18 13:32:31.028390 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.028357 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 13:32:31.086775 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:31.086734 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a9cf36e12cd844e88b9ba2474ad903.slice/crio-e93ed60465904d9a5df1c64a37a7b4afe2985eb5454be92c20bb4cc887ef4b0f WatchSource:0}: Error finding container e93ed60465904d9a5df1c64a37a7b4afe2985eb5454be92c20bb4cc887ef4b0f: Status 404 returned error can't find the container with id e93ed60465904d9a5df1c64a37a7b4afe2985eb5454be92c20bb4cc887ef4b0f Jul 18 13:32:31.092897 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.092878 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 13:32:31.115689 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:31.115666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e775caa402aa8a152d18579ae9d26d9.slice/crio-7b57afdf0fa53514dbf3286135239ca54b1f02dd0a7879f6e7049f64f0953878 WatchSource:0}: Error finding container 7b57afdf0fa53514dbf3286135239ca54b1f02dd0a7879f6e7049f64f0953878: Status 404 returned error can't find the container with id 7b57afdf0fa53514dbf3286135239ca54b1f02dd0a7879f6e7049f64f0953878 Jul 18 13:32:31.572742 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.572708 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 13:32:31.783719 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.783685 2564 apiserver.go:52] "Watching apiserver" Jul 18 13:32:31.792962 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.792756 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 18 13:32:31.793795 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.793763 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-r6pzv","openshift-network-diagnostics/network-check-target-225q5","openshift-network-operator/iptables-alerter-bhhwv","openshift-ovn-kubernetes/ovnkube-node-6rjnf","kube-system/global-pull-secret-syncer-ctdfr","kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cqll6","openshift-dns/node-resolver-gk8qd","openshift-multus/multus-additional-cni-plugins-bxqcb","openshift-multus/network-metrics-daemon-wbwm2","kube-system/konnectivity-agent-mvnm2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh","openshift-image-registry/node-ca-ms9b7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal"] Jul 18 13:32:31.796233 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.796210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.798912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.798950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.799022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2l6r4\"" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.799035 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.799115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.799163 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.799218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.799223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 18 13:32:31.802625 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.801420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.804557 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.804531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.805780 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.805568 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 18 13:32:31.805780 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.805654 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.805780 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.805714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:31.805978 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.805842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tbnc6\"" Jul 18 13:32:31.806126 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.806104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 18 13:32:31.806397 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.806300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 18 13:32:31.806688 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.806649 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.807416 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.807220 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.807484 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.807428 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 18 13:32:31.808286 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.807794 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vjh78\"" Jul 18 13:32:31.808286 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.807830 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 18 13:32:31.808286 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.808016 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.808286 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.808017 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.808286 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.808282 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 18 13:32:31.809832 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.809026 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.810778 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.810762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.813386 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.813352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.814171 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.814151 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.814297 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.814276 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.814428 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.814413 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.814538 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.814516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ms952\"" Jul 18 13:32:31.814751 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.814734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4c2sp\"" Jul 18 13:32:31.814826 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.814765 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.815795 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.815775 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxj27\"" Jul 18 13:32:31.816031 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.815980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:31.816099 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.816043 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:31.816406 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.816351 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 18 13:32:31.816582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.816566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 18 13:32:31.817843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.817825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:31.822630 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.821920 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 18 13:32:31.822630 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.822061 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 18 13:32:31.822630 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.822143 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vttz2\"" Jul 18 13:32:31.822828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.822717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-cni-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.822948 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.822924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-kubelet\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.823030 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.822970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j94p\" (UniqueName: \"kubernetes.io/projected/58e3f636-d6a9-4a7c-910f-991065c5b0e0-kube-api-access-8j94p\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-node-log\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-host\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-k8s-cni-cncf-io\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovnkube-config\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovnkube-script-lib\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-kubernetes\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysctl-d\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9t4\" (UniqueName: \"kubernetes.io/projected/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-kube-api-access-rf9t4\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.823978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysctl-conf\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-cnibin\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-kubelet\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-run\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-var-lib-kubelet\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-system-cni-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfkq\" (UniqueName: \"kubernetes.io/projected/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-kube-api-access-sdfkq\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-modprobe-d\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30288396-d289-4c60-a4d1-4da17345d284-tmp-dir\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-hostroot\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-etc-kubernetes\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-ovn\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-lib-modules\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58e3f636-d6a9-4a7c-910f-991065c5b0e0-cni-binary-copy\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-run-netns\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-env-overrides\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysconfig\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-tuned\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-netns\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-kubelet-config\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.824733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-sys\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc72d\" (UniqueName: \"kubernetes.io/projected/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-kube-api-access-nc72d\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-conf-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-daemon-config\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-systemd\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-host-slash\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-cni-multus\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-systemd-units\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-slash\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-cni-bin\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-cni-netd\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrtx\" (UniqueName: \"kubernetes.io/projected/0abdc362-e11f-4542-bf79-82fe21a8e65e-kube-api-access-xwrtx\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-socket-dir-parent\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-systemd\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-var-lib-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.825418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-os-release\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-cni-bin\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.824978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-multus-certs\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-etc-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-log-socket\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-dbus\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdc362-e11f-4542-bf79-82fe21a8e65e-tmp\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-iptables-alerter-script\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30288396-d289-4c60-a4d1-4da17345d284-hosts-file\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.825169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklbq\" (UniqueName: \"kubernetes.io/projected/30288396-d289-4c60-a4d1-4da17345d284-kube-api-access-nklbq\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.826341 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.826712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.828437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.827602 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:31.829509 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.829492 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.829766 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.829744 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.830145 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.830122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lhpm5\"" Jul 18 13:32:31.830416 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.830396 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 18 13:32:31.831528 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.830754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 18 13:32:31.831528 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.831013 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 18 13:32:31.831528 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.831238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lw8h8\"" Jul 18 13:32:31.831528 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.831437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 18 13:32:31.854716 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.854012 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5h5sn" Jul 18 13:32:31.864579 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.864557 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5h5sn" Jul 18 13:32:31.880197 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.880178 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 13:32:31.908915 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.908888 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 18 13:32:31.926331 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.926331 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovnkube-config\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovnkube-script-lib\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-kubernetes\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysctl-d\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2923f5c6-d83f-4b2f-a161-cd736a53bce4-serviceca\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9t4\" (UniqueName: \"kubernetes.io/projected/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-kube-api-access-rf9t4\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysctl-conf\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926562 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-cnibin\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-etc-selinux\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-kubelet\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-run\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-var-lib-kubelet\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926698 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/889d03f4-e9aa-4b62-bc1b-fd2f77093382-konnectivity-ca\") pod \"konnectivity-agent-mvnm2\" (UID: \"889d03f4-e9aa-4b62-bc1b-fd2f77093382\") " pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-system-cni-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfkq\" (UniqueName: \"kubernetes.io/projected/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-kube-api-access-sdfkq\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-modprobe-d\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30288396-d289-4c60-a4d1-4da17345d284-tmp-dir\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-hostroot\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-etc-kubernetes\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-ovn\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-lib-modules\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58e3f636-d6a9-4a7c-910f-991065c5b0e0-cni-binary-copy\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.926994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-run-netns\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.926997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-env-overrides\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovnkube-config\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysconfig\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-tuned\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq894\" (UniqueName: \"kubernetes.io/projected/68126ecb-8bc1-4c55-9e40-247dc92fa660-kube-api-access-sq894\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-netns\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-kubelet-config\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-sys\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc72d\" (UniqueName: \"kubernetes.io/projected/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-kube-api-access-nc72d\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-system-cni-dir\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-conf-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-daemon-config\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.927789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-systemd\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-host-slash\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-cni-multus\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-systemd-units\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-slash\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-cni-bin\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovnkube-script-lib\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-cni-netd\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrtx\" (UniqueName: \"kubernetes.io/projected/0abdc362-e11f-4542-bf79-82fe21a8e65e-kube-api-access-xwrtx\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/889d03f4-e9aa-4b62-bc1b-fd2f77093382-agent-certs\") pod \"konnectivity-agent-mvnm2\" (UID: \"889d03f4-e9aa-4b62-bc1b-fd2f77093382\") " pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-kubernetes\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-socket-dir-parent\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-systemd\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-var-lib-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cnibin\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysctl-d\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.928584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-os-release\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-cni-bin\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2923f5c6-d83f-4b2f-a161-cd736a53bce4-host\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-registration-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-multus-certs\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-etc-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-log-socket\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.927985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-dbus\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdc362-e11f-4542-bf79-82fe21a8e65e-tmp\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-iptables-alerter-script\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30288396-d289-4c60-a4d1-4da17345d284-hosts-file\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-system-cni-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nklbq\" (UniqueName: \"kubernetes.io/projected/30288396-d289-4c60-a4d1-4da17345d284-kube-api-access-nklbq\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-kubelet\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928232 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-run\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-var-lib-kubelet\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.929287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-modprobe-d\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928721 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30288396-d289-4c60-a4d1-4da17345d284-tmp-dir\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-hostroot\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-etc-kubernetes\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-ovn\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-iptables-alerter-script\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.928983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30288396-d289-4c60-a4d1-4da17345d284-hosts-file\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.929010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-lib-modules\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.929019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.929065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-run-netns\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.929470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-env-overrides\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.929541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysconfig\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.929666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58e3f636-d6a9-4a7c-910f-991065c5b0e0-cni-binary-copy\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-cni-bin\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930730 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-netns\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930730 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-kubelet-config\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.930730 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-sys\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.930730 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-conf-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930730 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-cni-bin\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930730 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930616 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-cni-netd\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-socket-dir-parent\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-daemon-config\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.930982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-run-systemd\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.930982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.930977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.931164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-var-lib-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.931164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-host-slash\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.931164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-var-lib-cni-multus\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.931164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-systemd\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.931164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-systemd-units\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.931164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-os-release\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-slash\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-cnibin\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.927934 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-sysctl-conf\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.931320 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:32:32.431290454 +0000 UTC m=+2.980855128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-dbus\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:31.931485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-log-socket\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-multus-certs\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-etc-openvswitch\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.930150 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:31.931613 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:32:32.431599779 +0000 UTC m=+2.981164439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-os-release\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-socket-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-device-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-cni-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.931790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-kubelet\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f8xc\" (UniqueName: \"kubernetes.io/projected/2923f5c6-d83f-4b2f-a161-cd736a53bce4-kube-api-access-7f8xc\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7w8c\" (UniqueName: \"kubernetes.io/projected/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-kube-api-access-r7w8c\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-multus-cni-dir\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j94p\" (UniqueName: \"kubernetes.io/projected/58e3f636-d6a9-4a7c-910f-991065c5b0e0-kube-api-access-8j94p\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-host-kubelet\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-node-log\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-host\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-sys-fs\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0abdc362-e11f-4542-bf79-82fe21a8e65e-host\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-node-log\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.931983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-k8s-cni-cncf-io\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.932296 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.932063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58e3f636-d6a9-4a7c-910f-991065c5b0e0-host-run-k8s-cni-cncf-io\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.932743 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.932542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.933986 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.933943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0abdc362-e11f-4542-bf79-82fe21a8e65e-etc-tuned\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.934125 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.934087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0abdc362-e11f-4542-bf79-82fe21a8e65e-tmp\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.947072 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.947027 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc72d\" (UniqueName: \"kubernetes.io/projected/dbfd86e8-1b94-4730-beb5-a7939f4b1cbc-kube-api-access-nc72d\") pod \"iptables-alerter-bhhwv\" (UID: \"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc\") " pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:31.947287 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.947085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9t4\" (UniqueName: \"kubernetes.io/projected/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-kube-api-access-rf9t4\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:31.947536 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.947424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfkq\" (UniqueName: \"kubernetes.io/projected/06575fd3-b79e-4112-a6a8-dbd3e5c22a4b-kube-api-access-sdfkq\") pod \"ovnkube-node-6rjnf\" (UID: \"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:31.947811 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.947762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklbq\" (UniqueName: \"kubernetes.io/projected/30288396-d289-4c60-a4d1-4da17345d284-kube-api-access-nklbq\") pod \"node-resolver-gk8qd\" (UID: \"30288396-d289-4c60-a4d1-4da17345d284\") " pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:31.949974 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.949947 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrtx\" (UniqueName: \"kubernetes.io/projected/0abdc362-e11f-4542-bf79-82fe21a8e65e-kube-api-access-xwrtx\") pod \"tuned-cqll6\" (UID: \"0abdc362-e11f-4542-bf79-82fe21a8e65e\") " pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:31.950424 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.950403 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j94p\" (UniqueName: \"kubernetes.io/projected/58e3f636-d6a9-4a7c-910f-991065c5b0e0-kube-api-access-8j94p\") pod \"multus-r6pzv\" (UID: \"58e3f636-d6a9-4a7c-910f-991065c5b0e0\") " pod="openshift-multus/multus-r6pzv" Jul 18 13:32:31.975044 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.974965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"5e775caa402aa8a152d18579ae9d26d9","Type":"ContainerStarted","Data":"7b57afdf0fa53514dbf3286135239ca54b1f02dd0a7879f6e7049f64f0953878"} Jul 18 13:32:31.976119 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:31.976089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"d7a9cf36e12cd844e88b9ba2474ad903","Type":"ContainerStarted","Data":"e93ed60465904d9a5df1c64a37a7b4afe2985eb5454be92c20bb4cc887ef4b0f"} Jul 18 13:32:32.032338 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-system-cni-dir\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032338 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-system-cni-dir\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.032584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/889d03f4-e9aa-4b62-bc1b-fd2f77093382-agent-certs\") pod \"konnectivity-agent-mvnm2\" (UID: \"889d03f4-e9aa-4b62-bc1b-fd2f77093382\") " pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:32.032584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cnibin\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2923f5c6-d83f-4b2f-a161-cd736a53bce4-host\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032590 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-registration-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cnibin\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-os-release\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2923f5c6-d83f-4b2f-a161-cd736a53bce4-host\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-registration-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-socket-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-os-release\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-device-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f8xc\" (UniqueName: \"kubernetes.io/projected/2923f5c6-d83f-4b2f-a161-cd736a53bce4-kube-api-access-7f8xc\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.032843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-device-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7w8c\" (UniqueName: \"kubernetes.io/projected/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-kube-api-access-r7w8c\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-sys-fs\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2923f5c6-d83f-4b2f-a161-cd736a53bce4-serviceca\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.032987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-socket-dir\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-sys-fs\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-etc-selinux\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/889d03f4-e9aa-4b62-bc1b-fd2f77093382-konnectivity-ca\") pod \"konnectivity-agent-mvnm2\" (UID: \"889d03f4-e9aa-4b62-bc1b-fd2f77093382\") " pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq894\" (UniqueName: \"kubernetes.io/projected/68126ecb-8bc1-4c55-9e40-247dc92fa660-kube-api-access-sq894\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68126ecb-8bc1-4c55-9e40-247dc92fa660-etc-selinux\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.033878 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.033878 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/889d03f4-e9aa-4b62-bc1b-fd2f77093382-konnectivity-ca\") pod \"konnectivity-agent-mvnm2\" (UID: \"889d03f4-e9aa-4b62-bc1b-fd2f77093382\") " pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:32.033878 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2923f5c6-d83f-4b2f-a161-cd736a53bce4-serviceca\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.033878 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.033843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.035764 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.035732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/889d03f4-e9aa-4b62-bc1b-fd2f77093382-agent-certs\") pod \"konnectivity-agent-mvnm2\" (UID: \"889d03f4-e9aa-4b62-bc1b-fd2f77093382\") " pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:32.040319 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.040276 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:32:32.040319 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.040306 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:32:32.040319 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.040323 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:32.040592 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.040422 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:32:32.540404279 +0000 UTC m=+3.089968941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:32.042980 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.042957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7w8c\" (UniqueName: \"kubernetes.io/projected/ee9705d3-8a04-41a6-8b60-eeeb0304ead9-kube-api-access-r7w8c\") pod \"multus-additional-cni-plugins-bxqcb\" (UID: \"ee9705d3-8a04-41a6-8b60-eeeb0304ead9\") " pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.043090 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.043024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq894\" (UniqueName: \"kubernetes.io/projected/68126ecb-8bc1-4c55-9e40-247dc92fa660-kube-api-access-sq894\") pod \"aws-ebs-csi-driver-node-xt7hh\" (UID: \"68126ecb-8bc1-4c55-9e40-247dc92fa660\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.044977 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.044951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f8xc\" (UniqueName: \"kubernetes.io/projected/2923f5c6-d83f-4b2f-a161-cd736a53bce4-kube-api-access-7f8xc\") pod \"node-ca-ms9b7\" (UID: \"2923f5c6-d83f-4b2f-a161-cd736a53bce4\") " pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.115318 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.115239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r6pzv" Jul 18 13:32:32.123926 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.123902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:32.137352 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.137328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bhhwv" Jul 18 13:32:32.142914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.142894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cqll6" Jul 18 13:32:32.150368 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.150345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gk8qd" Jul 18 13:32:32.160062 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.160039 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" Jul 18 13:32:32.167725 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.167706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:32.177298 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.177281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" Jul 18 13:32:32.182733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.182718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ms9b7" Jul 18 13:32:32.436898 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.436866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:32.437073 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.436918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:32.437073 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.437026 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:32.437191 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.437092 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:32:33.43707313 +0000 UTC m=+3.986637789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:32.437191 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.437028 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:32.437191 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.437140 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:32:33.437129443 +0000 UTC m=+3.986694104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:32.638215 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.638178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:32.638425 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.638390 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:32:32.638425 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.638414 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:32:32.638425 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.638427 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:32.638577 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.638489 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:32:33.63846995 +0000 UTC m=+4.188034625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:32.867831 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.867742 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 13:27:31 +0000 UTC" deadline="2028-01-27 00:23:45.016735259 +0000 UTC" Jul 18 13:32:32.867831 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.867776 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13378h51m12.148963994s" Jul 18 13:32:32.931269 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.931237 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e3f636_d6a9_4a7c_910f_991065c5b0e0.slice/crio-3d7a47b17d817ed06674d71c39e8b67fc307bb283cd774491050349f838eca49 WatchSource:0}: Error finding container 3d7a47b17d817ed06674d71c39e8b67fc307bb283cd774491050349f838eca49: Status 404 returned error can't find the container with id 3d7a47b17d817ed06674d71c39e8b67fc307bb283cd774491050349f838eca49 Jul 18 13:32:32.932611 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.932582 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06575fd3_b79e_4112_a6a8_dbd3e5c22a4b.slice/crio-2efcc1ef9e269a48f0b9e65aad6abc96a2c6710c3ed615f56bc7e303f98cd659 WatchSource:0}: Error finding container 2efcc1ef9e269a48f0b9e65aad6abc96a2c6710c3ed615f56bc7e303f98cd659: Status 404 returned error can't find the container with id 2efcc1ef9e269a48f0b9e65aad6abc96a2c6710c3ed615f56bc7e303f98cd659 Jul 18 13:32:32.935864 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.935806 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30288396_d289_4c60_a4d1_4da17345d284.slice/crio-c991120fcc8830482ff04c58ee78039ec0e6de6151b67549e8ab1735d38f9937 WatchSource:0}: Error finding container c991120fcc8830482ff04c58ee78039ec0e6de6151b67549e8ab1735d38f9937: Status 404 returned error can't find the container with id c991120fcc8830482ff04c58ee78039ec0e6de6151b67549e8ab1735d38f9937 Jul 18 13:32:32.936679 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.936657 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfd86e8_1b94_4730_beb5_a7939f4b1cbc.slice/crio-5df7bc96d1cbb383a897cbb26f310310ad078b42d7c43bb9f9b9998a533c4b22 WatchSource:0}: Error finding container 5df7bc96d1cbb383a897cbb26f310310ad078b42d7c43bb9f9b9998a533c4b22: Status 404 returned error can't find the container with id 5df7bc96d1cbb383a897cbb26f310310ad078b42d7c43bb9f9b9998a533c4b22 Jul 18 13:32:32.937624 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.937435 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68126ecb_8bc1_4c55_9e40_247dc92fa660.slice/crio-30e461b7da1c7f6a6dc74f09c5aa1e202b3b20b9af322d881d5563a4570234f1 WatchSource:0}: Error finding container 30e461b7da1c7f6a6dc74f09c5aa1e202b3b20b9af322d881d5563a4570234f1: Status 404 returned error can't find the container with id 30e461b7da1c7f6a6dc74f09c5aa1e202b3b20b9af322d881d5563a4570234f1 Jul 18 13:32:32.938805 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.938782 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889d03f4_e9aa_4b62_bc1b_fd2f77093382.slice/crio-2e65ed99aa2684369f7c20e186efda7a0bb4abc7290f360f14724ca34f568c64 WatchSource:0}: Error finding container 2e65ed99aa2684369f7c20e186efda7a0bb4abc7290f360f14724ca34f568c64: Status 404 returned error can't find the container with id 2e65ed99aa2684369f7c20e186efda7a0bb4abc7290f360f14724ca34f568c64 Jul 18 13:32:32.939223 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.939200 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abdc362_e11f_4542_bf79_82fe21a8e65e.slice/crio-2f661be16db02056accc323558751a42bb7e59c9d5ff84577c7b101af3154a0a WatchSource:0}: Error finding container 2f661be16db02056accc323558751a42bb7e59c9d5ff84577c7b101af3154a0a: Status 404 returned error can't find the container with id 2f661be16db02056accc323558751a42bb7e59c9d5ff84577c7b101af3154a0a Jul 18 13:32:32.941961 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.940448 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2923f5c6_d83f_4b2f_a161_cd736a53bce4.slice/crio-6a5f2d829a9ba2c3cd769f9e403d371ade6552b12715b9ff8c91357565fc3646 WatchSource:0}: Error finding container 6a5f2d829a9ba2c3cd769f9e403d371ade6552b12715b9ff8c91357565fc3646: Status 404 returned error can't find the container with id 6a5f2d829a9ba2c3cd769f9e403d371ade6552b12715b9ff8c91357565fc3646 Jul 18 13:32:32.942132 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:32:32.942003 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9705d3_8a04_41a6_8b60_eeeb0304ead9.slice/crio-c4a9dc47619f6e0eeb2eb16a723d25d5e013413d8d4eb7ef3cac5ea7d4f6e5bd WatchSource:0}: Error finding container c4a9dc47619f6e0eeb2eb16a723d25d5e013413d8d4eb7ef3cac5ea7d4f6e5bd: Status 404 returned error can't find the container with id c4a9dc47619f6e0eeb2eb16a723d25d5e013413d8d4eb7ef3cac5ea7d4f6e5bd Jul 18 13:32:32.966155 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.966129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:32.966258 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.966233 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:32.966302 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.966266 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:32.966340 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:32.966329 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:32.979542 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.979502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cqll6" event={"ID":"0abdc362-e11f-4542-bf79-82fe21a8e65e","Type":"ContainerStarted","Data":"2f661be16db02056accc323558751a42bb7e59c9d5ff84577c7b101af3154a0a"} Jul 18 13:32:32.982917 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.982884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvnm2" event={"ID":"889d03f4-e9aa-4b62-bc1b-fd2f77093382","Type":"ContainerStarted","Data":"2e65ed99aa2684369f7c20e186efda7a0bb4abc7290f360f14724ca34f568c64"} Jul 18 13:32:32.985346 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.985320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bhhwv" event={"ID":"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc","Type":"ContainerStarted","Data":"5df7bc96d1cbb383a897cbb26f310310ad078b42d7c43bb9f9b9998a533c4b22"} Jul 18 13:32:32.986965 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.986933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gk8qd" event={"ID":"30288396-d289-4c60-a4d1-4da17345d284","Type":"ContainerStarted","Data":"c991120fcc8830482ff04c58ee78039ec0e6de6151b67549e8ab1735d38f9937"} Jul 18 13:32:32.987905 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.987883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"2efcc1ef9e269a48f0b9e65aad6abc96a2c6710c3ed615f56bc7e303f98cd659"} Jul 18 13:32:32.988881 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.988855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerStarted","Data":"c4a9dc47619f6e0eeb2eb16a723d25d5e013413d8d4eb7ef3cac5ea7d4f6e5bd"} Jul 18 13:32:32.989803 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.989776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ms9b7" event={"ID":"2923f5c6-d83f-4b2f-a161-cd736a53bce4","Type":"ContainerStarted","Data":"6a5f2d829a9ba2c3cd769f9e403d371ade6552b12715b9ff8c91357565fc3646"} Jul 18 13:32:32.990794 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.990767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" event={"ID":"68126ecb-8bc1-4c55-9e40-247dc92fa660","Type":"ContainerStarted","Data":"30e461b7da1c7f6a6dc74f09c5aa1e202b3b20b9af322d881d5563a4570234f1"} Jul 18 13:32:32.991876 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:32.991857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r6pzv" event={"ID":"58e3f636-d6a9-4a7c-910f-991065c5b0e0","Type":"ContainerStarted","Data":"3d7a47b17d817ed06674d71c39e8b67fc307bb283cd774491050349f838eca49"} Jul 18 13:32:33.443269 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.442886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:33.443269 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.443172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:33.443515 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.443299 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:33.443515 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.443360 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:32:35.443342034 +0000 UTC m=+5.992906709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:33.443627 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.443563 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:33.443627 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.443609 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:32:35.443595112 +0000 UTC m=+5.993159778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:33.646148 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.645511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:33.646148 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.645696 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:32:33.646148 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.645716 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:32:33.646148 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.645729 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:33.646148 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.645789 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:32:35.645768679 +0000 UTC m=+6.195333341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:33.869125 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.868951 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 13:27:31 +0000 UTC" deadline="2028-03-02 18:18:00.430786482 +0000 UTC" Jul 18 13:32:33.869125 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.869012 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14236h45m26.561778065s" Jul 18 13:32:33.969351 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.969323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:33.969828 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:33.969467 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:33.998994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.998962 2564 generic.go:356] "Generic (PLEG): container finished" podID="5e775caa402aa8a152d18579ae9d26d9" containerID="77c31fd23d1f39ec9b2c9e83f7bea380c92c6bbb3d5cb547b7d3b1bed4ca4258" exitCode=0 Jul 18 13:32:33.999445 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:33.999419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"5e775caa402aa8a152d18579ae9d26d9","Type":"ContainerDied","Data":"77c31fd23d1f39ec9b2c9e83f7bea380c92c6bbb3d5cb547b7d3b1bed4ca4258"} Jul 18 13:32:34.014991 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:34.014945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"d7a9cf36e12cd844e88b9ba2474ad903","Type":"ContainerStarted","Data":"290822acfed0ea87ee5f79596d839226c74793217118a1136cb55e432105941c"} Jul 18 13:32:34.032116 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:34.031440 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" podStartSLOduration=3.031418352 podStartE2EDuration="3.031418352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:34.03085987 +0000 UTC m=+4.580424554" watchObservedRunningTime="2026-07-18 13:32:34.031418352 +0000 UTC m=+4.580983038" Jul 18 13:32:34.967981 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:34.967267 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:34.967981 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:34.967416 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:34.967981 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:34.967805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:34.967981 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:34.967913 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:35.033911 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:35.033444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"5e775caa402aa8a152d18579ae9d26d9","Type":"ContainerStarted","Data":"df7f63dd73e30ca9acfc33281fed0e9e367ae7252249004684d43b58788b013a"} Jul 18 13:32:35.461004 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:35.460966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:35.461179 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:35.461030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:35.461237 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.461187 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:35.461302 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.461253 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:32:39.461233255 +0000 UTC m=+10.010797916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:35.461833 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.461693 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:35.461833 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.461752 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:32:39.461736528 +0000 UTC m=+10.011301200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:35.666313 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:35.666270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:35.666503 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.666476 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:32:35.666503 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.666503 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:32:35.666630 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.666516 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:35.666630 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.666583 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:32:39.666565856 +0000 UTC m=+10.216130519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:35.967177 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:35.966693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:35.967177 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:35.966833 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:36.967210 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:36.967088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:36.967210 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:36.967140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:36.967751 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:36.967230 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:36.967751 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:36.967690 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:37.966220 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:37.966185 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:37.966428 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:37.966328 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:38.966558 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:38.966523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:38.967004 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:38.966664 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:38.967004 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:38.966679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:38.967004 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:38.966805 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:39.500116 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:39.500065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:39.500319 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:39.500131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:39.500319 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.500265 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:39.500458 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.500328 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:32:47.500312599 +0000 UTC m=+18.049877258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:39.500740 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.500641 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:39.500740 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.500717 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:32:47.500700783 +0000 UTC m=+18.050265458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:39.701875 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:39.701843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:39.702046 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.701986 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:32:39.702046 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.702002 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:32:39.702046 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.702011 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:39.702193 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.702060 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:32:47.702047617 +0000 UTC m=+18.251612275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:39.967826 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:39.967791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:39.968297 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:39.967907 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:40.966903 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:40.966867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:40.967075 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:40.967008 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:40.967141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:40.967082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:40.967215 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:40.967182 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:41.967390 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:41.967346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:41.967863 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:41.967489 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:42.966299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:42.966267 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:42.966495 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:42.966269 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:42.966495 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:42.966418 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:42.966602 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:42.966513 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:43.966445 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:43.966397 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:43.966930 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:43.966543 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:44.967030 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:44.966999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:44.967030 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:44.967016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:44.967591 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:44.967126 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:44.967591 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:44.967274 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:45.967006 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:45.966975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:45.967186 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:45.967114 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:46.966556 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:46.966514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:46.966774 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:46.966518 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:46.966774 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:46.966637 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:46.966774 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:46.966732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:47.561638 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:47.561597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:47.562159 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:47.561669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:47.562159 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.561784 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:47.562159 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.561787 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:47.562159 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.561858 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:33:03.561837504 +0000 UTC m=+34.111402177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:32:47.562159 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.561876 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:33:03.561866816 +0000 UTC m=+34.111431476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:32:47.764193 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:47.764147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:47.764398 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.764297 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:32:47.764398 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.764327 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:32:47.764398 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.764342 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:47.764545 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.764428 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:33:03.764405077 +0000 UTC m=+34.313969739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:32:47.966922 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:47.966886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:47.967121 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:47.966992 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:48.966387 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:48.966341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:48.966387 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:48.966387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:48.966841 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:48.966495 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:48.966841 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:48.966628 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:49.966940 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:49.966905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:49.967425 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:49.967000 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:50.966798 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:50.966761 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:50.966982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:50.966774 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:50.966982 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:50.966907 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:50.966982 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:50.966969 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:51.966392 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:51.966348 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:51.966585 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:51.966475 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:52.967503 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:52.967115 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:52.968445 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:52.967135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:52.968445 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:52.967618 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:52.968445 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:52.967642 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:53.073022 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.072984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cqll6" event={"ID":"0abdc362-e11f-4542-bf79-82fe21a8e65e","Type":"ContainerStarted","Data":"5971c5c6bf272c0fb2dd964b1eea0d9994d82175f136bff816d5f21cc5a00f92"} Jul 18 13:32:53.074268 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.074240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvnm2" event={"ID":"889d03f4-e9aa-4b62-bc1b-fd2f77093382","Type":"ContainerStarted","Data":"d1ba06bcd34c6417e4edce161273bb8b138d07273e59a276b3a5cc72a6063dd9"} Jul 18 13:32:53.075428 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.075403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gk8qd" event={"ID":"30288396-d289-4c60-a4d1-4da17345d284","Type":"ContainerStarted","Data":"d855dc539698530efc5f310079705d34e661f8b99f29f145960c9ee70d33a249"} Jul 18 13:32:53.078288 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.078264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"2400547b6a05009822fa0b38ca20875c15dcc20ac40cfa3831c8943ea6ab42d4"} Jul 18 13:32:53.078409 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.078326 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"79de8cb20f9484ee4f382212cbf8bc28120dfd85cb7763d647fb7d928418a108"} Jul 18 13:32:53.078409 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.078357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"3b6720388377735438ba7891754de22060cf22a8cf30f857e9243b60494b110b"} Jul 18 13:32:53.078504 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.078407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"a0f6b1a9ff22944f38fd1d576a945c69726eab356975a1416894dabd062cd5fe"} Jul 18 13:32:53.078504 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.078454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"89b26f215b30d749347084836e8bb37f0109314537e0248682bc93bacc53e431"} Jul 18 13:32:53.078594 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.078502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"58ead79012869cdbea9496ce7822655792ccc6318f45a960066ea5fcd36d38a3"} Jul 18 13:32:53.079762 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.079741 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee9705d3-8a04-41a6-8b60-eeeb0304ead9" containerID="4ba8e626c100f05940358925931306431a7f215c8f3b8019fa73cd79b2882384" exitCode=0 Jul 18 13:32:53.079843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.079806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerDied","Data":"4ba8e626c100f05940358925931306431a7f215c8f3b8019fa73cd79b2882384"} Jul 18 13:32:53.081299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.081279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ms9b7" event={"ID":"2923f5c6-d83f-4b2f-a161-cd736a53bce4","Type":"ContainerStarted","Data":"ebb6be52ce9cbd3ad4b2e47b4dedff4504d09b81c706d696f6fbc38d5ad5aa6b"} Jul 18 13:32:53.082809 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.082790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" event={"ID":"68126ecb-8bc1-4c55-9e40-247dc92fa660","Type":"ContainerStarted","Data":"55c3ecd98a619a7d942563d591164dcfc7199b4b8394b5b792c4afbe0a4bc367"} Jul 18 13:32:53.084147 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.084129 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r6pzv" event={"ID":"58e3f636-d6a9-4a7c-910f-991065c5b0e0","Type":"ContainerStarted","Data":"25bb467c33f2a382b0daacc380bde2cbb02b17cb1700cee6910822bd1962b42a"} Jul 18 13:32:53.090792 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.090751 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cqll6" podStartSLOduration=4.034765469 podStartE2EDuration="23.090740379s" totalImagesPullingTime="19.05597491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:53.090598697 +0000 UTC m=+23.640163379" watchObservedRunningTime="2026-07-18 13:32:53.090740379 +0000 UTC m=+23.640305060" Jul 18 13:32:53.091094 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.091070 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" podStartSLOduration=22.091065639 podStartE2EDuration="22.091065639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:35.048782614 +0000 UTC m=+5.598347297" watchObservedRunningTime="2026-07-18 13:32:53.091065639 +0000 UTC m=+23.640630320" Jul 18 13:32:53.116104 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.116057 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mvnm2" podStartSLOduration=12.287985075 podStartE2EDuration="23.116042967s" totalImagesPullingTime="10.828057892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:53.103599565 +0000 UTC m=+23.653164245" watchObservedRunningTime="2026-07-18 13:32:53.116042967 +0000 UTC m=+23.665607648" Jul 18 13:32:53.116260 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.116187 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ms9b7" podStartSLOduration=4.063127854 podStartE2EDuration="23.116180348s" totalImagesPullingTime="19.053052494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:53.115978406 +0000 UTC m=+23.665543088" watchObservedRunningTime="2026-07-18 13:32:53.116180348 +0000 UTC m=+23.665745063" Jul 18 13:32:53.131898 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.131851 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r6pzv" podStartSLOduration=5.05040395 podStartE2EDuration="24.131837414s" totalImagesPullingTime="19.081433464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:53.131307018 +0000 UTC m=+23.680871698" watchObservedRunningTime="2026-07-18 13:32:53.131837414 +0000 UTC m=+23.681402126" Jul 18 13:32:53.164278 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.164234 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gk8qd" podStartSLOduration=4.106451402 podStartE2EDuration="23.1642185s" totalImagesPullingTime="19.057767098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:53.163666132 +0000 UTC m=+23.713230818" watchObservedRunningTime="2026-07-18 13:32:53.1642185 +0000 UTC m=+23.713783181" Jul 18 13:32:53.471671 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.471644 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 18 13:32:53.879948 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.879831 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T13:32:53.471665649Z","UUID":"f8df0ebd-a940-469a-a0b2-dc622e34cd86","Handler":null,"Name":"","Endpoint":""} Jul 18 13:32:53.881715 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.881685 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 18 13:32:53.881845 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.881733 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 18 13:32:53.966955 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:53.966924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:53.967147 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:53.967062 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:54.087559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:54.087515 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bhhwv" event={"ID":"dbfd86e8-1b94-4730-beb5-a7939f4b1cbc","Type":"ContainerStarted","Data":"190da612ee0c1d599706218e3eb00aa18aeed230af6141e7575daf6b11bdcbff"} Jul 18 13:32:54.089605 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:54.089568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" event={"ID":"68126ecb-8bc1-4c55-9e40-247dc92fa660","Type":"ContainerStarted","Data":"ec9888e2e6ff50bc4f3b204b7dee8ac776c76d158c2625ce1cd8ac89b7f927d7"} Jul 18 13:32:54.103719 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:54.103662 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bhhwv" podStartSLOduration=5.047023899 podStartE2EDuration="24.103641233s" totalImagesPullingTime="19.056617334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:54.10299984 +0000 UTC m=+24.652564522" watchObservedRunningTime="2026-07-18 13:32:54.103641233 +0000 UTC m=+24.653205916" Jul 18 13:32:54.966320 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:54.966128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:54.966509 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:54.966128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:54.966509 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:54.966434 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:54.966596 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:54.966518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:55.096367 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:55.096277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"17efb94850f83556d3b71a84f796c3f2438df76c71eff1a99e7b99477c4bdef5"} Jul 18 13:32:55.099864 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:55.099816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" event={"ID":"68126ecb-8bc1-4c55-9e40-247dc92fa660","Type":"ContainerStarted","Data":"7eec1a5ad0ca9b6e89816abc132dcd51903221cf39564d3faa68ab06ba7d831a"} Jul 18 13:32:55.118216 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:55.118161 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xt7hh" podStartSLOduration=3.801608286 podStartE2EDuration="25.118139816s" totalImagesPullingTime="21.31653153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:55.118047556 +0000 UTC m=+25.667612239" watchObservedRunningTime="2026-07-18 13:32:55.118139816 +0000 UTC m=+25.667704498" Jul 18 13:32:55.384745 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:55.384664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:55.385339 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:55.385319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:55.966619 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:55.966585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:55.966810 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:55.966688 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:56.102040 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:56.102005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:56.102625 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:56.102605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mvnm2" Jul 18 13:32:56.966625 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:56.966595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:56.966798 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:56.966593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:56.966798 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:56.966740 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:56.966908 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:56.966821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:57.966445 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:57.966137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:57.967367 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:57.966555 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:58.110337 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.110232 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" event={"ID":"06575fd3-b79e-4112-a6a8-dbd3e5c22a4b","Type":"ContainerStarted","Data":"7e0c36752f70b0be6b82afff2827fef69bac6c0c44c28cfa9e5e18382dcc08b5"} Jul 18 13:32:58.110636 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.110604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:58.112152 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.112122 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee9705d3-8a04-41a6-8b60-eeeb0304ead9" containerID="baeb4339387b6adb1a5c3bc10b410722af9144f6a8a404c17dcd10ec51514fd9" exitCode=0 Jul 18 13:32:58.112319 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.112197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerDied","Data":"baeb4339387b6adb1a5c3bc10b410722af9144f6a8a404c17dcd10ec51514fd9"} Jul 18 13:32:58.126559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.126536 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:58.137622 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.137574 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" podStartSLOduration=8.948602837 podStartE2EDuration="28.137560921s" totalImagesPullingTime="19.188958084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:32:58.137319299 +0000 UTC m=+28.686883979" watchObservedRunningTime="2026-07-18 13:32:58.137560921 +0000 UTC m=+28.687125601" Jul 18 13:32:58.966957 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.966920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:58.967464 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:58.966926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:58.967464 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:58.967027 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:58.967464 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:58.967149 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:32:59.114396 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.114356 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:59.114610 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.114457 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:59.136168 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.136143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:32:59.578886 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.578798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ctdfr"] Jul 18 13:32:59.579048 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.578939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:32:59.579103 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:59.579055 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:32:59.581864 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.581837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-225q5"] Jul 18 13:32:59.581997 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.581947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:32:59.582087 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:59.582064 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:32:59.590147 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.590119 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbwm2"] Jul 18 13:32:59.590254 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:32:59.590220 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:32:59.590362 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:32:59.590322 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:33:00.119407 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:00.119361 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee9705d3-8a04-41a6-8b60-eeeb0304ead9" containerID="67effcbaedd2701d49be7b25dba47b854aa2555f06259e1551d423f9efa580a3" exitCode=0 Jul 18 13:33:00.119818 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:00.119474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerDied","Data":"67effcbaedd2701d49be7b25dba47b854aa2555f06259e1551d423f9efa580a3"} Jul 18 13:33:00.966256 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:00.966227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:00.966444 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:00.966332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:00.966444 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:00.966360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:33:00.966541 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:00.966338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:33:00.969546 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:00.966994 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:33:00.969546 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:00.967147 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:33:02.126750 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:02.126713 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee9705d3-8a04-41a6-8b60-eeeb0304ead9" containerID="5c0a09358294ea6b83d568f395a346c34a1ca76e69f5b4f65981fcffde997d31" exitCode=0 Jul 18 13:33:02.126750 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:02.126755 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerDied","Data":"5c0a09358294ea6b83d568f395a346c34a1ca76e69f5b4f65981fcffde997d31"} Jul 18 13:33:02.966858 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:02.966826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:02.967026 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:02.966969 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:33:02.967101 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:02.967066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:02.967153 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:02.967101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:33:02.967202 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:02.967180 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:33:02.967305 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:02.967285 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:33:03.582597 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:03.582557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:03.583093 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:03.582627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:33:03.583093 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.582721 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 13:33:03.583093 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.582769 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:33:03.583093 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.582806 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret podName:bf9097cb-304f-4a1a-85f3-f60bd96eddc9 nodeName:}" failed. No retries permitted until 2026-07-18 13:33:35.582786477 +0000 UTC m=+66.132351147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret") pod "global-pull-secret-syncer-ctdfr" (UID: "bf9097cb-304f-4a1a-85f3-f60bd96eddc9") : object "kube-system"/"original-pull-secret" not registered Jul 18 13:33:03.583093 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.582825 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:33:35.58281555 +0000 UTC m=+66.132380210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 13:33:03.784567 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:03.784530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:03.784760 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.784661 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 13:33:03.784760 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.784686 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 13:33:03.784760 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.784699 2564 projected.go:196] Error preparing data for projected volume kube-api-access-mzc2v for pod openshift-network-diagnostics/network-check-target-225q5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:33:03.784927 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:03.784770 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v podName:1775449e-bebd-4ca6-bf7b-30353584ac0c nodeName:}" failed. No retries permitted until 2026-07-18 13:33:35.784744028 +0000 UTC m=+66.334308689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzc2v" (UniqueName: "kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v") pod "network-check-target-225q5" (UID: "1775449e-bebd-4ca6-bf7b-30353584ac0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 13:33:04.966492 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:04.966455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:04.966928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:04.966457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:04.966928 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:04.966568 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-225q5" podUID="1775449e-bebd-4ca6-bf7b-30353584ac0c" Jul 18 13:33:04.966928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:04.966467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:33:04.966928 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:04.966645 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ctdfr" podUID="bf9097cb-304f-4a1a-85f3-f60bd96eddc9" Jul 18 13:33:04.966928 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:04.966746 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:33:05.787684 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.787611 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeReady" Jul 18 13:33:05.787846 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.787765 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 18 13:33:05.827963 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.827927 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k9pcn"] Jul 18 13:33:05.830453 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.830427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:05.830873 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.830846 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jmz7z"] Jul 18 13:33:05.832549 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.832531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:05.833037 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.833011 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 18 13:33:05.833143 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.833067 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t94bf\"" Jul 18 13:33:05.833143 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.833067 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 18 13:33:05.835059 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.835037 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 18 13:33:05.835169 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.835105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 18 13:33:05.835283 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.835266 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v6fm2\"" Jul 18 13:33:05.835283 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.835280 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 18 13:33:05.843114 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.843089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9pcn"] Jul 18 13:33:05.843941 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:05.843890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmz7z"] Jul 18 13:33:06.000613 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.000576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnd64\" (UniqueName: \"kubernetes.io/projected/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-kube-api-access-fnd64\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.001141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.000640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.001141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.000717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwb7w\" (UniqueName: \"kubernetes.io/projected/31e00d6b-d536-4001-8049-cb466a6257df-kube-api-access-wwb7w\") pod \"ingress-canary-jmz7z\" (UID: \"31e00d6b-d536-4001-8049-cb466a6257df\") " pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.001141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.000779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31e00d6b-d536-4001-8049-cb466a6257df-cert\") pod \"ingress-canary-jmz7z\" (UID: \"31e00d6b-d536-4001-8049-cb466a6257df\") " pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.001141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.000809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-config-volume\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.001141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.000832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-tmp-dir\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.102017 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.101924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.102017 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.101979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwb7w\" (UniqueName: \"kubernetes.io/projected/31e00d6b-d536-4001-8049-cb466a6257df-kube-api-access-wwb7w\") pod \"ingress-canary-jmz7z\" (UID: \"31e00d6b-d536-4001-8049-cb466a6257df\") " pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.102299 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:06.102091 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:06.102299 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:06.102162 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:33:06.602138742 +0000 UTC m=+37.151703403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:06.102299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.102203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31e00d6b-d536-4001-8049-cb466a6257df-cert\") pod \"ingress-canary-jmz7z\" (UID: \"31e00d6b-d536-4001-8049-cb466a6257df\") " pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.102299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.102231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-config-volume\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.102299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.102254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-tmp-dir\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.102577 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.102316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnd64\" (UniqueName: \"kubernetes.io/projected/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-kube-api-access-fnd64\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.102743 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.102712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-tmp-dir\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.102930 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.102906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-config-volume\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.106943 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.106916 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31e00d6b-d536-4001-8049-cb466a6257df-cert\") pod \"ingress-canary-jmz7z\" (UID: \"31e00d6b-d536-4001-8049-cb466a6257df\") " pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.111061 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.111024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnd64\" (UniqueName: \"kubernetes.io/projected/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-kube-api-access-fnd64\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.111173 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.111114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwb7w\" (UniqueName: \"kubernetes.io/projected/31e00d6b-d536-4001-8049-cb466a6257df-kube-api-access-wwb7w\") pod \"ingress-canary-jmz7z\" (UID: \"31e00d6b-d536-4001-8049-cb466a6257df\") " pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.149787 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.149756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmz7z" Jul 18 13:33:06.407664 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.407425 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmz7z"] Jul 18 13:33:06.412720 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:33:06.412676 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e00d6b_d536_4001_8049_cb466a6257df.slice/crio-65fb35e086955b2dc85163d26976af093c8524c247948cb807f6a68c34ac26ec WatchSource:0}: Error finding container 65fb35e086955b2dc85163d26976af093c8524c247948cb807f6a68c34ac26ec: Status 404 returned error can't find the container with id 65fb35e086955b2dc85163d26976af093c8524c247948cb807f6a68c34ac26ec Jul 18 13:33:06.605943 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.605902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:06.606106 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:06.606042 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:06.606106 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:06.606103 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:33:07.606088934 +0000 UTC m=+38.155653596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:06.966334 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.966305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:06.966590 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.966305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:33:06.966590 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.966305 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:06.970690 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.970664 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 13:33:06.970690 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.970688 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 13:33:06.970863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.970664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 13:33:06.970863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.970757 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 13:33:06.970863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.970668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5j6b\"" Jul 18 13:33:06.970863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:06.970668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfhtr\"" Jul 18 13:33:07.144684 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:07.144634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmz7z" event={"ID":"31e00d6b-d536-4001-8049-cb466a6257df","Type":"ContainerStarted","Data":"65fb35e086955b2dc85163d26976af093c8524c247948cb807f6a68c34ac26ec"} Jul 18 13:33:07.148060 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:07.148030 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee9705d3-8a04-41a6-8b60-eeeb0304ead9" containerID="75ac02a2875ba1871f4c7125216ce7f6d24c43645f83cd9ca5b96487226d459c" exitCode=0 Jul 18 13:33:07.148198 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:07.148075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerDied","Data":"75ac02a2875ba1871f4c7125216ce7f6d24c43645f83cd9ca5b96487226d459c"} Jul 18 13:33:07.613497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:07.613408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:07.613623 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:07.613584 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:07.613679 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:07.613648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:33:09.613631991 +0000 UTC m=+40.163196649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:08.154091 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:08.153883 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee9705d3-8a04-41a6-8b60-eeeb0304ead9" containerID="800e054c78b4982633f8d7fed68ebbbcd19a5e3c213d132dcc47aa8ac7fb779a" exitCode=0 Jul 18 13:33:08.154500 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:08.153920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerDied","Data":"800e054c78b4982633f8d7fed68ebbbcd19a5e3c213d132dcc47aa8ac7fb779a"} Jul 18 13:33:09.158593 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:09.158551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmz7z" event={"ID":"31e00d6b-d536-4001-8049-cb466a6257df","Type":"ContainerStarted","Data":"1da5f43d5de5d2914610b9b7db40c073c6a4723712784f9b453ca43147b82087"} Jul 18 13:33:09.162126 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:09.162095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" event={"ID":"ee9705d3-8a04-41a6-8b60-eeeb0304ead9","Type":"ContainerStarted","Data":"b30ef8fc5d9a52a90583a8b13b863d3c33c9d5ffde4a17c71f538073c9684d2e"} Jul 18 13:33:09.174155 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:09.174099 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jmz7z" podStartSLOduration=1.877486698 podStartE2EDuration="4.174082451s" totalImagesPullingTime="2.296595753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:33:09.172901309 +0000 UTC m=+39.722465990" watchObservedRunningTime="2026-07-18 13:33:09.174082451 +0000 UTC m=+39.723647132" Jul 18 13:33:09.193809 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:09.193744 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bxqcb" podStartSLOduration=8.989105605 podStartE2EDuration="39.193719953s" totalImagesPullingTime="30.204614348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:33:09.19151603 +0000 UTC m=+39.741080711" watchObservedRunningTime="2026-07-18 13:33:09.193719953 +0000 UTC m=+39.743284634" Jul 18 13:33:09.629288 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:09.629252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:09.629514 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:09.629421 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:09.629514 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:09.629485 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:33:13.629469118 +0000 UTC m=+44.179033777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:13.656458 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:13.656419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:13.656850 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:13.656531 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:13.656850 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:13.656600 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:33:21.656583467 +0000 UTC m=+52.206148127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:21.709188 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:21.709137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:21.709678 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:21.709269 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:21.709678 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:21.709323 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:33:37.709309233 +0000 UTC m=+68.258873893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:31.138327 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:31.138296 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rjnf" Jul 18 13:33:35.604125 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.604073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:35.604125 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.604131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:33:35.606942 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.606919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 13:33:35.607033 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.606943 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 13:33:35.615195 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:35.615171 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 18 13:33:35.615271 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:35.615233 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:34:39.61521719 +0000 UTC m=+130.164781852 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : secret "metrics-daemon-secret" not found Jul 18 13:33:35.619019 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.618994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf9097cb-304f-4a1a-85f3-f60bd96eddc9-original-pull-secret\") pod \"global-pull-secret-syncer-ctdfr\" (UID: \"bf9097cb-304f-4a1a-85f3-f60bd96eddc9\") " pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:35.778133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.778096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ctdfr" Jul 18 13:33:35.806701 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.806669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:35.809622 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.809599 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 13:33:35.820186 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.820159 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 13:33:35.830807 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.830769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzc2v\" (UniqueName: \"kubernetes.io/projected/1775449e-bebd-4ca6-bf7b-30353584ac0c-kube-api-access-mzc2v\") pod \"network-check-target-225q5\" (UID: \"1775449e-bebd-4ca6-bf7b-30353584ac0c\") " pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:35.898881 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:35.898845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ctdfr"] Jul 18 13:33:35.901707 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:33:35.901677 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9097cb_304f_4a1a_85f3_f60bd96eddc9.slice/crio-1147b12ac33bf36a3245a4edd8d6fa9dc46783e669ca4d1d06d9bae5a55a105e WatchSource:0}: Error finding container 1147b12ac33bf36a3245a4edd8d6fa9dc46783e669ca4d1d06d9bae5a55a105e: Status 404 returned error can't find the container with id 1147b12ac33bf36a3245a4edd8d6fa9dc46783e669ca4d1d06d9bae5a55a105e Jul 18 13:33:36.092884 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:36.092852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfhtr\"" Jul 18 13:33:36.101036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:36.101018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:36.218302 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:36.218260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-225q5"] Jul 18 13:33:36.221934 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:33:36.221906 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1775449e_bebd_4ca6_bf7b_30353584ac0c.slice/crio-73d54c3feee88d7282257bfb67d4f51f55b1888462ef4f9edaf4681d49b1e5fe WatchSource:0}: Error finding container 73d54c3feee88d7282257bfb67d4f51f55b1888462ef4f9edaf4681d49b1e5fe: Status 404 returned error can't find the container with id 73d54c3feee88d7282257bfb67d4f51f55b1888462ef4f9edaf4681d49b1e5fe Jul 18 13:33:36.224566 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:36.224534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ctdfr" event={"ID":"bf9097cb-304f-4a1a-85f3-f60bd96eddc9","Type":"ContainerStarted","Data":"1147b12ac33bf36a3245a4edd8d6fa9dc46783e669ca4d1d06d9bae5a55a105e"} Jul 18 13:33:37.228493 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:37.228458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-225q5" event={"ID":"1775449e-bebd-4ca6-bf7b-30353584ac0c","Type":"ContainerStarted","Data":"73d54c3feee88d7282257bfb67d4f51f55b1888462ef4f9edaf4681d49b1e5fe"} Jul 18 13:33:37.721739 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:37.721697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:33:37.721903 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:37.721881 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:33:37.721976 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:33:37.721962 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:34:09.721940954 +0000 UTC m=+100.271505615 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:33:41.242391 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:41.242342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ctdfr" event={"ID":"bf9097cb-304f-4a1a-85f3-f60bd96eddc9","Type":"ContainerStarted","Data":"31dd62379a24c4457abe39273439467b25afb585915ae1fc54a8031a7ea51767"} Jul 18 13:33:41.243859 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:41.243830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-225q5" event={"ID":"1775449e-bebd-4ca6-bf7b-30353584ac0c","Type":"ContainerStarted","Data":"c1f3437e8cad42cdf1a46583ecf8b38a54d2de12edb89d41e18b63d7de06ce4d"} Jul 18 13:33:41.244010 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:41.243963 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:33:41.257699 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:41.257647 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ctdfr" podStartSLOduration=66.33332187 podStartE2EDuration="1m11.257628683s" totalImagesPullingTime="4.924306813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:33:41.257205114 +0000 UTC m=+71.806769796" watchObservedRunningTime="2026-07-18 13:33:41.257628683 +0000 UTC m=+71.807193364" Jul 18 13:33:41.272424 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:33:41.272360 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-225q5" podStartSLOduration=66.669464183 podStartE2EDuration="1m11.272345167s" totalImagesPullingTime="4.602880984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:33:41.271006018 +0000 UTC m=+71.820570699" watchObservedRunningTime="2026-07-18 13:33:41.272345167 +0000 UTC m=+71.821909848" Jul 18 13:34:05.521276 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.521165 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss"] Jul 18 13:34:05.523742 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.523724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.527661 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.527635 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 18 13:34:05.528446 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.528421 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 18 13:34:05.528569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.528454 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 18 13:34:05.528569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.528464 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 18 13:34:05.528569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.528464 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 18 13:34:05.528569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.528468 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 18 13:34:05.528821 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.528798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 18 13:34:05.533570 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.533551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss"] Jul 18 13:34:05.611321 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.611282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhvn\" (UniqueName: \"kubernetes.io/projected/f0160101-a0b4-4a99-bffc-833782c0cb2d-kube-api-access-7jhvn\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.611321 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.611321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-ca\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.611548 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.611344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.611548 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.611364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.611548 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.611407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0160101-a0b4-4a99-bffc-833782c0cb2d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.611548 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.611440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-hub\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.712663 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.712628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-ca\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.712760 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.712673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.712760 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.712698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.712760 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.712723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0160101-a0b4-4a99-bffc-833782c0cb2d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.712863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.712780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-hub\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.712896 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.712870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhvn\" (UniqueName: \"kubernetes.io/projected/f0160101-a0b4-4a99-bffc-833782c0cb2d-kube-api-access-7jhvn\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.713582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.713559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0160101-a0b4-4a99-bffc-833782c0cb2d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.715254 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.715220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.715368 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.715350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.715523 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.715506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-ca\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.715747 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.715728 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f0160101-a0b4-4a99-bffc-833782c0cb2d-hub\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.721477 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.721453 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhvn\" (UniqueName: \"kubernetes.io/projected/f0160101-a0b4-4a99-bffc-833782c0cb2d-kube-api-access-7jhvn\") pod \"cluster-proxy-proxy-agent-76b6f765bd-lpcss\" (UID: \"f0160101-a0b4-4a99-bffc-833782c0cb2d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.849034 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.848942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:34:05.974639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:05.974601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss"] Jul 18 13:34:05.977291 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:34:05.977260 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0160101_a0b4_4a99_bffc_833782c0cb2d.slice/crio-e469a395d5381c2e3fb681c1e0fa7fba049ffeca38510bb560a039f3c3cadcec WatchSource:0}: Error finding container e469a395d5381c2e3fb681c1e0fa7fba049ffeca38510bb560a039f3c3cadcec: Status 404 returned error can't find the container with id e469a395d5381c2e3fb681c1e0fa7fba049ffeca38510bb560a039f3c3cadcec Jul 18 13:34:06.304092 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:06.304051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" event={"ID":"f0160101-a0b4-4a99-bffc-833782c0cb2d","Type":"ContainerStarted","Data":"e469a395d5381c2e3fb681c1e0fa7fba049ffeca38510bb560a039f3c3cadcec"} Jul 18 13:34:09.741138 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:09.741114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:34:09.741467 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:34:09.741227 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:34:09.741467 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:34:09.741279 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:35:13.741265543 +0000 UTC m=+164.290830201 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:34:10.317198 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:10.317160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" event={"ID":"f0160101-a0b4-4a99-bffc-833782c0cb2d","Type":"ContainerStarted","Data":"b74b4cd8e730e1cb92ad0c2d6c2c8ddb69288aff16599d2f944f3443b48f9a05"} Jul 18 13:34:10.317198 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:10.317206 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" event={"ID":"f0160101-a0b4-4a99-bffc-833782c0cb2d","Type":"ContainerStarted","Data":"e98dc5f7ffc1d2173f0f9c6036deda741f1e33ed1d29fba89b0d615107742561"} Jul 18 13:34:10.317434 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:10.317219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" event={"ID":"f0160101-a0b4-4a99-bffc-833782c0cb2d","Type":"ContainerStarted","Data":"c6bf728dac14480cd798172092f89be54824b5fe9c52e80eacffc9b97d27e882"} Jul 18 13:34:10.336189 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:10.336140 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" podStartSLOduration=1.6879487050000002 podStartE2EDuration="5.336122438s" totalImagesPullingTime="3.648173733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:34:10.335773036 +0000 UTC m=+100.885337728" watchObservedRunningTime="2026-07-18 13:34:10.336122438 +0000 UTC m=+100.885687117" Jul 18 13:34:12.249322 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:12.249288 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-225q5" Jul 18 13:34:39.645011 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:34:39.644970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:34:39.645457 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:34:39.645130 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 18 13:34:39.645457 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:34:39.645210 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs podName:8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe nodeName:}" failed. No retries permitted until 2026-07-18 13:36:41.645192586 +0000 UTC m=+252.194757244 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs") pod "network-metrics-daemon-wbwm2" (UID: "8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe") : secret "metrics-daemon-secret" not found Jul 18 13:35:08.843607 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:35:08.843559 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-k9pcn" podUID="2e9d48b8-6612-4efd-9c5c-34bbaa70457e" Jul 18 13:35:09.473315 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:35:09.473291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9pcn" Jul 18 13:35:09.985437 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:35:09.985390 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wbwm2" podUID="8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe" Jul 18 13:35:13.777506 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:35:13.777460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:35:13.777880 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:35:13.777574 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 13:35:13.777880 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:35:13.777625 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls podName:2e9d48b8-6612-4efd-9c5c-34bbaa70457e nodeName:}" failed. No retries permitted until 2026-07-18 13:37:15.777610822 +0000 UTC m=+286.327175481 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls") pod "dns-default-k9pcn" (UID: "2e9d48b8-6612-4efd-9c5c-34bbaa70457e") : secret "dns-default-metrics-tls" not found Jul 18 13:35:23.966839 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:35:23.966758 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:36:22.494854 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.494815 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4"] Jul 18 13:36:22.497576 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.497559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.500260 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.500236 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 18 13:36:22.500401 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.500236 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 18 13:36:22.500401 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.500288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 18 13:36:22.500401 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.500299 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7tmks\"" Jul 18 13:36:22.501252 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.501235 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 18 13:36:22.510622 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.510602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4"] Jul 18 13:36:22.649423 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.649345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f1d947a-076d-472b-ad74-073668f6f035-config\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.649605 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.649439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f1d947a-076d-472b-ad74-073668f6f035-serving-cert\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.649605 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.649494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pbg\" (UniqueName: \"kubernetes.io/projected/2f1d947a-076d-472b-ad74-073668f6f035-kube-api-access-j5pbg\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.750145 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.750053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f1d947a-076d-472b-ad74-073668f6f035-config\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.750145 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.750097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f1d947a-076d-472b-ad74-073668f6f035-serving-cert\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.750145 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.750132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pbg\" (UniqueName: \"kubernetes.io/projected/2f1d947a-076d-472b-ad74-073668f6f035-kube-api-access-j5pbg\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.750657 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.750627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f1d947a-076d-472b-ad74-073668f6f035-config\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.752347 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.752328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f1d947a-076d-472b-ad74-073668f6f035-serving-cert\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.757751 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.757731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pbg\" (UniqueName: \"kubernetes.io/projected/2f1d947a-076d-472b-ad74-073668f6f035-kube-api-access-j5pbg\") pod \"service-ca-operator-5bcbc94c78-x6cl4\" (UID: \"2f1d947a-076d-472b-ad74-073668f6f035\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.806095 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.806058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" Jul 18 13:36:22.929317 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:22.929286 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4"] Jul 18 13:36:22.932647 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:36:22.932616 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1d947a_076d_472b_ad74_073668f6f035.slice/crio-9c6507729cf27a621e59db25f8903911fb2404bec28f614668007ca7b034e248 WatchSource:0}: Error finding container 9c6507729cf27a621e59db25f8903911fb2404bec28f614668007ca7b034e248: Status 404 returned error can't find the container with id 9c6507729cf27a621e59db25f8903911fb2404bec28f614668007ca7b034e248 Jul 18 13:36:23.679957 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:23.679919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" event={"ID":"2f1d947a-076d-472b-ad74-073668f6f035","Type":"ContainerStarted","Data":"9c6507729cf27a621e59db25f8903911fb2404bec28f614668007ca7b034e248"} Jul 18 13:36:25.694275 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:25.694232 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" event={"ID":"2f1d947a-076d-472b-ad74-073668f6f035","Type":"ContainerStarted","Data":"83f7b73296db04ef5ee54c637eebf9b0578a26cb89597974f7fe5c5c9d37c1bb"} Jul 18 13:36:25.715442 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:25.715370 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" podStartSLOduration=1.8991138269999999 podStartE2EDuration="3.715355966s" totalImagesPullingTime="1.816242139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:36:25.714067697 +0000 UTC m=+236.263632378" watchObservedRunningTime="2026-07-18 13:36:25.715355966 +0000 UTC m=+236.264920647" Jul 18 13:36:28.057550 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.057516 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-25r94"] Jul 18 13:36:28.060731 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.060712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.063554 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.063531 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 18 13:36:28.063680 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.063538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 18 13:36:28.063680 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.063538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 18 13:36:28.063680 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.063608 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 18 13:36:28.064557 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.064536 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pprp\"" Jul 18 13:36:28.071475 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.071455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-25r94"] Jul 18 13:36:28.193976 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.193939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.193976 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.193979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.194234 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.194000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789m4\" (UniqueName: \"kubernetes.io/projected/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-api-access-789m4\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.194234 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.194086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ceebf852-8a47-433c-b42a-b0191bcb04ac-crio-socket\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.194234 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.194123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ceebf852-8a47-433c-b42a-b0191bcb04ac-data-volume\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.294812 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.294777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ceebf852-8a47-433c-b42a-b0191bcb04ac-crio-socket\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.294812 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.294812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ceebf852-8a47-433c-b42a-b0191bcb04ac-data-volume\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.295019 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.294872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.295019 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.294902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.295019 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.294904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ceebf852-8a47-433c-b42a-b0191bcb04ac-crio-socket\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.295019 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.294965 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 13:36:28.295276 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.295014 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 13:36:28.295276 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.295032 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:28.795018712 +0000 UTC m=+239.344583375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : configmap "kube-rbac-proxy" not found Jul 18 13:36:28.295276 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.295046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789m4\" (UniqueName: \"kubernetes.io/projected/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-api-access-789m4\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.295276 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.295070 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:28.795057011 +0000 UTC m=+239.344621670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : secret "insights-runtime-extractor-tls" not found Jul 18 13:36:28.295276 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.295173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ceebf852-8a47-433c-b42a-b0191bcb04ac-data-volume\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.305526 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.305501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-789m4\" (UniqueName: \"kubernetes.io/projected/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-api-access-789m4\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.798667 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.798634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.798667 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:28.798674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:28.798888 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.798765 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 13:36:28.798888 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.798798 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 13:36:28.798888 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.798840 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:29.798823241 +0000 UTC m=+240.348387902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : configmap "kube-rbac-proxy" not found Jul 18 13:36:28.798888 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:28.798856 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:29.798849026 +0000 UTC m=+240.348413684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : secret "insights-runtime-extractor-tls" not found Jul 18 13:36:29.808554 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:29.808521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:29.808964 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:29.808566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:29.808964 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:29.808605 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 13:36:29.808964 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:29.808682 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 13:36:29.808964 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:29.808697 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:31.808674556 +0000 UTC m=+242.358239215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : configmap "kube-rbac-proxy" not found Jul 18 13:36:29.808964 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:29.808721 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:31.808710455 +0000 UTC m=+242.358275115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : secret "insights-runtime-extractor-tls" not found Jul 18 13:36:31.039611 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:31.039581 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gk8qd_30288396-d289-4c60-a4d1-4da17345d284/dns-node-resolver/0.log" Jul 18 13:36:31.824950 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:31.824910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:31.824950 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:31.824953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:31.825176 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:31.825031 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 13:36:31.825176 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:31.825052 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 13:36:31.825176 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:31.825094 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:35.825078063 +0000 UTC m=+246.374642722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : configmap "kube-rbac-proxy" not found Jul 18 13:36:31.825176 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:31.825108 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:35.825101776 +0000 UTC m=+246.374666434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : secret "insights-runtime-extractor-tls" not found Jul 18 13:36:32.043294 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:32.043267 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ms9b7_2923f5c6-d83f-4b2f-a161-cd736a53bce4/node-ca/0.log" Jul 18 13:36:32.840159 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:32.840080 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmz7z_31e00d6b-d536-4001-8049-cb466a6257df/serve-healthcheck-canary/0.log" Jul 18 13:36:35.854962 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:35.854935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:35.855342 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:35.854970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:35.855342 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:35.855049 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 13:36:35.855342 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:35.855103 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 13:36:35.855342 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:35.855116 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:43.855099585 +0000 UTC m=+254.404664244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : configmap "kube-rbac-proxy" not found Jul 18 13:36:35.855342 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:35.855141 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls podName:ceebf852-8a47-433c-b42a-b0191bcb04ac nodeName:}" failed. No retries permitted until 2026-07-18 13:36:43.8551304 +0000 UTC m=+254.404695062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-25r94" (UID: "ceebf852-8a47-433c-b42a-b0191bcb04ac") : secret "insights-runtime-extractor-tls" not found Jul 18 13:36:41.696069 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:41.696008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:36:41.698440 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:41.698415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe-metrics-certs\") pod \"network-metrics-daemon-wbwm2\" (UID: \"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe\") " pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:36:41.969650 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:41.969572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5j6b\"" Jul 18 13:36:41.977649 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:41.977631 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbwm2" Jul 18 13:36:42.104009 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:42.103976 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbwm2"] Jul 18 13:36:42.107021 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:36:42.106994 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb91a5c_b9f8_42cc_85d6_dbcbef7336fe.slice/crio-03a05d2a194151d354afb3ebdbf7758e61575354d885e2f807d97126ea4337c2 WatchSource:0}: Error finding container 03a05d2a194151d354afb3ebdbf7758e61575354d885e2f807d97126ea4337c2: Status 404 returned error can't find the container with id 03a05d2a194151d354afb3ebdbf7758e61575354d885e2f807d97126ea4337c2 Jul 18 13:36:42.746001 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:42.745964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbwm2" event={"ID":"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe","Type":"ContainerStarted","Data":"03a05d2a194151d354afb3ebdbf7758e61575354d885e2f807d97126ea4337c2"} Jul 18 13:36:43.750466 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.750431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbwm2" event={"ID":"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe","Type":"ContainerStarted","Data":"55bb9e41100c6c24713eaa01ee08d36d8ea510d59b6c49ffd1d491e51897e009"} Jul 18 13:36:43.750835 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.750483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbwm2" event={"ID":"8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe","Type":"ContainerStarted","Data":"fb0c18b723c65da64410ddcdc1ce578580f1e0069d4822a0a57fe799407dac9a"} Jul 18 13:36:43.766984 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.766907 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wbwm2" podStartSLOduration=253.711638564 podStartE2EDuration="4m14.766890521s" totalImagesPullingTime="1.055251957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:36:43.765407855 +0000 UTC m=+254.314972536" watchObservedRunningTime="2026-07-18 13:36:43.766890521 +0000 UTC m=+254.316455202" Jul 18 13:36:43.912024 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.911986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:43.912024 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.912025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:43.912622 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.912596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ceebf852-8a47-433c-b42a-b0191bcb04ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:43.914400 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.914361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ceebf852-8a47-433c-b42a-b0191bcb04ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-25r94\" (UID: \"ceebf852-8a47-433c-b42a-b0191bcb04ac\") " pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:43.973464 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.971523 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pprp\"" Jul 18 13:36:43.979127 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:43.979107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-25r94" Jul 18 13:36:44.100429 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:44.100393 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-25r94"] Jul 18 13:36:44.104117 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:36:44.104089 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceebf852_8a47_433c_b42a_b0191bcb04ac.slice/crio-b41d3877fbfb07786e0c89ec26ee0b0f75f7177820fd384fd6895d34dd0d1f53 WatchSource:0}: Error finding container b41d3877fbfb07786e0c89ec26ee0b0f75f7177820fd384fd6895d34dd0d1f53: Status 404 returned error can't find the container with id b41d3877fbfb07786e0c89ec26ee0b0f75f7177820fd384fd6895d34dd0d1f53 Jul 18 13:36:44.757231 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:44.756941 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25r94" event={"ID":"ceebf852-8a47-433c-b42a-b0191bcb04ac","Type":"ContainerStarted","Data":"b41d3877fbfb07786e0c89ec26ee0b0f75f7177820fd384fd6895d34dd0d1f53"} Jul 18 13:36:48.301567 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.301529 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78b5b6cc4c-vc4xx"] Jul 18 13:36:48.308266 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.308241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.311415 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.311385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 18 13:36:48.312182 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.312159 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 18 13:36:48.313063 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.313041 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xfjwq\"" Jul 18 13:36:48.318162 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.318139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 18 13:36:48.332838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.332813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 18 13:36:48.344742 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.344714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b5b6cc4c-vc4xx"] Jul 18 13:36:48.348828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.348798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5jx\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-kube-api-access-6f5jx\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.348979 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.348863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a4b88f-fa98-401e-9698-74492b1da989-ca-trust-extracted\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.348979 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.348934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a4b88f-fa98-401e-9698-74492b1da989-trusted-ca\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.349099 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.348976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a4b88f-fa98-401e-9698-74492b1da989-image-registry-private-configuration\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.349099 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.349010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a4b88f-fa98-401e-9698-74492b1da989-registry-certificates\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.349099 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.349058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a4b88f-fa98-401e-9698-74492b1da989-installation-pull-secrets\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.349241 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.349103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-registry-tls\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.349241 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.349130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-bound-sa-token\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a4b88f-fa98-401e-9698-74492b1da989-ca-trust-extracted\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a4b88f-fa98-401e-9698-74492b1da989-trusted-ca\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a4b88f-fa98-401e-9698-74492b1da989-image-registry-private-configuration\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a4b88f-fa98-401e-9698-74492b1da989-registry-certificates\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a4b88f-fa98-401e-9698-74492b1da989-installation-pull-secrets\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-registry-tls\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.449914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-bound-sa-token\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.450202 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.449970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5jx\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-kube-api-access-6f5jx\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.450202 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.450039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a4b88f-fa98-401e-9698-74492b1da989-ca-trust-extracted\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.450724 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.450699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a4b88f-fa98-401e-9698-74492b1da989-registry-certificates\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.450843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.450806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a4b88f-fa98-401e-9698-74492b1da989-trusted-ca\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.452253 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.452227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a4b88f-fa98-401e-9698-74492b1da989-image-registry-private-configuration\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.452614 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.452588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-registry-tls\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.452898 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.452882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a4b88f-fa98-401e-9698-74492b1da989-installation-pull-secrets\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.458805 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.458778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-bound-sa-token\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.459598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.459577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5jx\" (UniqueName: \"kubernetes.io/projected/76a4b88f-fa98-401e-9698-74492b1da989-kube-api-access-6f5jx\") pod \"image-registry-78b5b6cc4c-vc4xx\" (UID: \"76a4b88f-fa98-401e-9698-74492b1da989\") " pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.619583 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.619494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:48.919452 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:48.919428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b5b6cc4c-vc4xx"] Jul 18 13:36:48.921483 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:36:48.921461 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a4b88f_fa98_401e_9698_74492b1da989.slice/crio-1d5e507e1275000b470b5efcf9f2613f0d8941ff6254fb0f48c60fd5471c3cab WatchSource:0}: Error finding container 1d5e507e1275000b470b5efcf9f2613f0d8941ff6254fb0f48c60fd5471c3cab: Status 404 returned error can't find the container with id 1d5e507e1275000b470b5efcf9f2613f0d8941ff6254fb0f48c60fd5471c3cab Jul 18 13:36:49.777602 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:49.777546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" event={"ID":"76a4b88f-fa98-401e-9698-74492b1da989","Type":"ContainerStarted","Data":"94f7601b96791ddb696d377f78844b8f66d9fb2fb44598dd31bd0642723485a9"} Jul 18 13:36:49.778066 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:49.777612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" event={"ID":"76a4b88f-fa98-401e-9698-74492b1da989","Type":"ContainerStarted","Data":"1d5e507e1275000b470b5efcf9f2613f0d8941ff6254fb0f48c60fd5471c3cab"} Jul 18 13:36:49.778066 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:49.777638 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:36:49.779256 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:49.779223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25r94" event={"ID":"ceebf852-8a47-433c-b42a-b0191bcb04ac","Type":"ContainerStarted","Data":"4a8533333de87a0fcddd3d2f560906706ac885e0934260f0ff70739924739420"} Jul 18 13:36:49.801816 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:49.801715 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" podStartSLOduration=1.801692973 podStartE2EDuration="1.801692973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:36:49.800617829 +0000 UTC m=+260.350182511" watchObservedRunningTime="2026-07-18 13:36:49.801692973 +0000 UTC m=+260.351257636" Jul 18 13:36:52.792129 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:52.792094 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25r94" event={"ID":"ceebf852-8a47-433c-b42a-b0191bcb04ac","Type":"ContainerStarted","Data":"e5181e1e1e7c414529fcff56f0887461e7ac7e5eadb3adb97f2337d043edbd5d"} Jul 18 13:36:55.803605 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:55.803566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-25r94" event={"ID":"ceebf852-8a47-433c-b42a-b0191bcb04ac","Type":"ContainerStarted","Data":"d0c304dc13605a1c70c61cfe454788df93782063059e8a61b5dede4ecf8a4428"} Jul 18 13:36:55.820981 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:55.820932 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-25r94" podStartSLOduration=16.385817054 podStartE2EDuration="27.820918089s" totalImagesPullingTime="11.435101035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 13:36:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:36:55.819793061 +0000 UTC m=+266.369357744" watchObservedRunningTime="2026-07-18 13:36:55.820918089 +0000 UTC m=+266.370482770" Jul 18 13:36:58.760948 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.760915 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95"] Jul 18 13:36:58.764060 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.764038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:36:58.766510 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.766490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2z2cg\"" Jul 18 13:36:58.766603 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.766490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 18 13:36:58.771766 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.771743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95"] Jul 18 13:36:58.834161 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.834125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/47ebf2ff-7da4-43dd-b4cb-3298f2963d3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7lx95\" (UID: \"47ebf2ff-7da4-43dd-b4cb-3298f2963d3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:36:58.935171 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:58.935133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/47ebf2ff-7da4-43dd-b4cb-3298f2963d3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7lx95\" (UID: \"47ebf2ff-7da4-43dd-b4cb-3298f2963d3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:36:58.935338 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:58.935252 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 18 13:36:58.935338 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:36:58.935308 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ebf2ff-7da4-43dd-b4cb-3298f2963d3a-tls-certificates podName:47ebf2ff-7da4-43dd-b4cb-3298f2963d3a nodeName:}" failed. No retries permitted until 2026-07-18 13:36:59.43529326 +0000 UTC m=+269.984857919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/47ebf2ff-7da4-43dd-b4cb-3298f2963d3a-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-7lx95" (UID: "47ebf2ff-7da4-43dd-b4cb-3298f2963d3a") : secret "prometheus-operator-admission-webhook-tls" not found Jul 18 13:36:59.439512 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:59.439479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/47ebf2ff-7da4-43dd-b4cb-3298f2963d3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7lx95\" (UID: \"47ebf2ff-7da4-43dd-b4cb-3298f2963d3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:36:59.441803 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:59.441775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/47ebf2ff-7da4-43dd-b4cb-3298f2963d3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7lx95\" (UID: \"47ebf2ff-7da4-43dd-b4cb-3298f2963d3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:36:59.674070 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:59.674023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:36:59.795804 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:59.795769 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95"] Jul 18 13:36:59.798456 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:36:59.798422 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ebf2ff_7da4_43dd_b4cb_3298f2963d3a.slice/crio-6c42ce43251c5d0224db58a07a530d7f39ac2685ab91c121113307f026f5475c WatchSource:0}: Error finding container 6c42ce43251c5d0224db58a07a530d7f39ac2685ab91c121113307f026f5475c: Status 404 returned error can't find the container with id 6c42ce43251c5d0224db58a07a530d7f39ac2685ab91c121113307f026f5475c Jul 18 13:36:59.816540 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:36:59.816512 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" event={"ID":"47ebf2ff-7da4-43dd-b4cb-3298f2963d3a","Type":"ContainerStarted","Data":"6c42ce43251c5d0224db58a07a530d7f39ac2685ab91c121113307f026f5475c"} Jul 18 13:37:01.827881 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:01.827843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" event={"ID":"47ebf2ff-7da4-43dd-b4cb-3298f2963d3a","Type":"ContainerStarted","Data":"24d085bf35616b58d59c5401a4af2c2edfb79744729a74b8b8ad4e7d5b8e9f6f"} Jul 18 13:37:01.828308 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:01.828099 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:37:01.832588 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:01.832562 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" Jul 18 13:37:01.843510 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:01.843457 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7lx95" podStartSLOduration=2.683529027 podStartE2EDuration="3.843443268s" totalImagesPullingTime="1.159914241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:01.842873408 +0000 UTC m=+272.392438090" watchObservedRunningTime="2026-07-18 13:37:01.843443268 +0000 UTC m=+272.393007948" Jul 18 13:37:02.577388 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.577352 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f88c6797-dmk7f"] Jul 18 13:37:02.580358 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.580340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.582987 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.582956 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 18 13:37:02.582987 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.582971 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 18 13:37:02.583154 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.583009 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 18 13:37:02.584151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.584130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 18 13:37:02.584151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.584143 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 18 13:37:02.584332 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.584231 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 18 13:37:02.584332 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.584295 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh5bg\"" Jul 18 13:37:02.584443 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.584369 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 18 13:37:02.589097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.589079 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 18 13:37:02.589798 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.589777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f88c6797-dmk7f"] Jul 18 13:37:02.663367 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-oauth-serving-cert\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.663367 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-oauth-config\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.663582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-serving-cert\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.663582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hs7\" (UniqueName: \"kubernetes.io/projected/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-kube-api-access-s2hs7\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.663582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-trusted-ca-bundle\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.663582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-config\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.663582 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.663545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-service-ca\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.764859 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.764806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-config\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.764859 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.764848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-service-ca\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765066 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-oauth-serving-cert\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765102 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-oauth-config\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765146 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-serving-cert\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765182 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hs7\" (UniqueName: \"kubernetes.io/projected/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-kube-api-access-s2hs7\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765235 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-trusted-ca-bundle\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765725 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-service-ca\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765725 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-config\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.765903 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.765732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-oauth-serving-cert\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.766077 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.766046 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-trusted-ca-bundle\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.767691 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.767672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-serving-cert\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.767782 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.767712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-oauth-config\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.775549 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.775529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hs7\" (UniqueName: \"kubernetes.io/projected/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-kube-api-access-s2hs7\") pod \"console-75f88c6797-dmk7f\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:02.890176 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:02.890079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:03.018455 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:03.018431 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f88c6797-dmk7f"] Jul 18 13:37:03.020894 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:37:03.020868 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667bb59a_aa47_4dc7_8060_4e3f24ce4ec0.slice/crio-a53a35ec63690331536f0fb217e0f5fcfd06286b392883a8f05be0374186ed0a WatchSource:0}: Error finding container a53a35ec63690331536f0fb217e0f5fcfd06286b392883a8f05be0374186ed0a: Status 404 returned error can't find the container with id a53a35ec63690331536f0fb217e0f5fcfd06286b392883a8f05be0374186ed0a Jul 18 13:37:03.836232 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:03.836189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f88c6797-dmk7f" event={"ID":"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0","Type":"ContainerStarted","Data":"a53a35ec63690331536f0fb217e0f5fcfd06286b392883a8f05be0374186ed0a"} Jul 18 13:37:05.844492 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:05.844455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f88c6797-dmk7f" event={"ID":"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0","Type":"ContainerStarted","Data":"2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc"} Jul 18 13:37:05.863359 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:05.863300 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75f88c6797-dmk7f" podStartSLOduration=1.219543163 podStartE2EDuration="3.863276318s" totalImagesPullingTime="2.643733155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:05.862615937 +0000 UTC m=+276.412180619" watchObservedRunningTime="2026-07-18 13:37:05.863276318 +0000 UTC m=+276.412841000" Jul 18 13:37:06.173023 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.172987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6"] Jul 18 13:37:06.176206 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.176187 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.178912 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.178885 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 18 13:37:06.179038 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.178889 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 18 13:37:06.179038 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.179008 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 18 13:37:06.179038 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.178959 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-czck2\"" Jul 18 13:37:06.179151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.178893 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 18 13:37:06.179151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.179036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 18 13:37:06.188765 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.188740 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6"] Jul 18 13:37:06.216753 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.216723 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cjhrl"] Jul 18 13:37:06.219950 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.219930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.222402 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.222363 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 18 13:37:06.222516 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.222458 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 18 13:37:06.222516 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.222493 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 18 13:37:06.222734 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.222720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zg879\"" Jul 18 13:37:06.294028 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.293987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2gz\" (UniqueName: \"kubernetes.io/projected/46213551-ca76-4a31-b772-31bc75914bf6-kube-api-access-5w2gz\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294209 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-root\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294209 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fde10a60-5f25-41d1-8845-8fcc4de3798d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.294209 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-tls\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294209 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd76w\" (UniqueName: \"kubernetes.io/projected/fde10a60-5f25-41d1-8845-8fcc4de3798d-kube-api-access-xd76w\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.294209 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde10a60-5f25-41d1-8845-8fcc4de3798d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.294439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-wtmp\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-accelerators-collector-config\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-textfile\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-sys\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46213551-ca76-4a31-b772-31bc75914bf6-metrics-client-ca\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.294618 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.294461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde10a60-5f25-41d1-8845-8fcc4de3798d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.394893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.394858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-tls\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.394893 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.394896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd76w\" (UniqueName: \"kubernetes.io/projected/fde10a60-5f25-41d1-8845-8fcc4de3798d-kube-api-access-xd76w\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.394917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde10a60-5f25-41d1-8845-8fcc4de3798d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.394950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-wtmp\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.394968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-accelerators-collector-config\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.394988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-textfile\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:37:06.395025 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-sys\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:37:06.395104 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-tls podName:46213551-ca76-4a31-b772-31bc75914bf6 nodeName:}" failed. No retries permitted until 2026-07-18 13:37:06.895083495 +0000 UTC m=+277.444648175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-tls") pod "node-exporter-cjhrl" (UID: "46213551-ca76-4a31-b772-31bc75914bf6") : secret "node-exporter-tls" not found Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-sys\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395133 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-wtmp\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46213551-ca76-4a31-b772-31bc75914bf6-metrics-client-ca\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde10a60-5f25-41d1-8845-8fcc4de3798d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.395639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2gz\" (UniqueName: \"kubernetes.io/projected/46213551-ca76-4a31-b772-31bc75914bf6-kube-api-access-5w2gz\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-root\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fde10a60-5f25-41d1-8845-8fcc4de3798d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.395891 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-accelerators-collector-config\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395891 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46213551-ca76-4a31-b772-31bc75914bf6-metrics-client-ca\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395891 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46213551-ca76-4a31-b772-31bc75914bf6-root\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.395891 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.395828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde10a60-5f25-41d1-8845-8fcc4de3798d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.396091 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.396012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-textfile\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.397678 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.397655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fde10a60-5f25-41d1-8845-8fcc4de3798d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.397891 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.397867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.398000 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.397983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fde10a60-5f25-41d1-8845-8fcc4de3798d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.403249 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.403224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd76w\" (UniqueName: \"kubernetes.io/projected/fde10a60-5f25-41d1-8845-8fcc4de3798d-kube-api-access-xd76w\") pod \"openshift-state-metrics-79f6958cdf-npfs6\" (UID: \"fde10a60-5f25-41d1-8845-8fcc4de3798d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.406725 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.406704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2gz\" (UniqueName: \"kubernetes.io/projected/46213551-ca76-4a31-b772-31bc75914bf6-kube-api-access-5w2gz\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.484856 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.484818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" Jul 18 13:37:06.618837 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.618656 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6"] Jul 18 13:37:06.621229 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:37:06.621204 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde10a60_5f25_41d1_8845_8fcc4de3798d.slice/crio-0c00f214ad2762885f34ce6eaa9d53726c555eaa723fa76ce550b35288d03ea8 WatchSource:0}: Error finding container 0c00f214ad2762885f34ce6eaa9d53726c555eaa723fa76ce550b35288d03ea8: Status 404 returned error can't find the container with id 0c00f214ad2762885f34ce6eaa9d53726c555eaa723fa76ce550b35288d03ea8 Jul 18 13:37:06.849416 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.849359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" event={"ID":"fde10a60-5f25-41d1-8845-8fcc4de3798d","Type":"ContainerStarted","Data":"45772b3e2b9e5eb2052b15ff85fda2df798cd339942894bb171ba1992c443946"} Jul 18 13:37:06.849864 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.849449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" event={"ID":"fde10a60-5f25-41d1-8845-8fcc4de3798d","Type":"ContainerStarted","Data":"7d7fe57437df550cf4181daf15c6f039b587fe1f69383e8f7f8ec8c5e869c863"} Jul 18 13:37:06.849864 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.849483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" event={"ID":"fde10a60-5f25-41d1-8845-8fcc4de3798d","Type":"ContainerStarted","Data":"0c00f214ad2762885f34ce6eaa9d53726c555eaa723fa76ce550b35288d03ea8"} Jul 18 13:37:06.898751 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.898706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-tls\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:06.901042 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:06.901014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46213551-ca76-4a31-b772-31bc75914bf6-node-exporter-tls\") pod \"node-exporter-cjhrl\" (UID: \"46213551-ca76-4a31-b772-31bc75914bf6\") " pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:07.128888 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:07.128803 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cjhrl" Jul 18 13:37:07.139886 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:37:07.139859 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46213551_ca76_4a31_b772_31bc75914bf6.slice/crio-ba50b1b465205f073ca62ed852feef80dde57d6ddb9da52e82ab5ef6d2d11477 WatchSource:0}: Error finding container ba50b1b465205f073ca62ed852feef80dde57d6ddb9da52e82ab5ef6d2d11477: Status 404 returned error can't find the container with id ba50b1b465205f073ca62ed852feef80dde57d6ddb9da52e82ab5ef6d2d11477 Jul 18 13:37:07.856787 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:07.856757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjhrl" event={"ID":"46213551-ca76-4a31-b772-31bc75914bf6","Type":"ContainerStarted","Data":"ba50b1b465205f073ca62ed852feef80dde57d6ddb9da52e82ab5ef6d2d11477"} Jul 18 13:37:08.624850 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:08.624772 2564 patch_prober.go:28] interesting pod/image-registry-78b5b6cc4c-vc4xx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 18 13:37:08.624850 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:08.624821 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" podUID="76a4b88f-fa98-401e-9698-74492b1da989" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 13:37:08.862367 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:08.862334 2564 generic.go:356] "Generic (PLEG): container finished" podID="46213551-ca76-4a31-b772-31bc75914bf6" containerID="fee71fa962fa0615283516a7672a726149669e7a4491d984485e5a2d679e0498" exitCode=0 Jul 18 13:37:08.862789 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:08.862431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjhrl" event={"ID":"46213551-ca76-4a31-b772-31bc75914bf6","Type":"ContainerDied","Data":"fee71fa962fa0615283516a7672a726149669e7a4491d984485e5a2d679e0498"} Jul 18 13:37:08.864400 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:08.864358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" event={"ID":"fde10a60-5f25-41d1-8845-8fcc4de3798d","Type":"ContainerStarted","Data":"33bfb686d8b97d604d3fa15c853836ce2a8d31ac4f889c6b18666ecb26112db7"} Jul 18 13:37:08.895665 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:08.895611 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-npfs6" podStartSLOduration=1.8254250490000001 podStartE2EDuration="2.895593851s" totalImagesPullingTime="1.070168802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:08.895013747 +0000 UTC m=+279.444578438" watchObservedRunningTime="2026-07-18 13:37:08.895593851 +0000 UTC m=+279.445158532" Jul 18 13:37:09.870091 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:09.870049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjhrl" event={"ID":"46213551-ca76-4a31-b772-31bc75914bf6","Type":"ContainerStarted","Data":"b701090d35cc87010eb45982baaeae86154db6a894dac9b2350e7a542ad32779"} Jul 18 13:37:09.870484 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:09.870152 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cjhrl" event={"ID":"46213551-ca76-4a31-b772-31bc75914bf6","Type":"ContainerStarted","Data":"f112c861b2c2b6712ff311c2bf532d9d02f253cb0a0584b12e546b79c43ae623"} Jul 18 13:37:09.890028 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:09.889976 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cjhrl" podStartSLOduration=2.712548145 podStartE2EDuration="3.889961266s" totalImagesPullingTime="1.114592035s" totalInitContainerRuntime="62.821086ms" isStatefulPod=true podCreationTimestamp="2026-07-18 13:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:09.887939387 +0000 UTC m=+280.437504069" watchObservedRunningTime="2026-07-18 13:37:09.889961266 +0000 UTC m=+280.439525986" Jul 18 13:37:10.459346 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.459302 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f9c5745b-bmd45"] Jul 18 13:37:10.462386 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.462349 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.464946 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.464923 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 18 13:37:10.466179 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.466151 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 18 13:37:10.466293 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.466190 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-10h7ur2g23av8\"" Jul 18 13:37:10.466293 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.466212 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 18 13:37:10.466293 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.466212 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 18 13:37:10.466293 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.466157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4cfn6\"" Jul 18 13:37:10.471271 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.471245 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f9c5745b-bmd45"] Jul 18 13:37:10.534295 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thftc\" (UniqueName: \"kubernetes.io/projected/93549ecb-87e4-4307-9066-1e6a03b62ad0-kube-api-access-thftc\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.534519 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93549ecb-87e4-4307-9066-1e6a03b62ad0-audit-log\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.534519 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93549ecb-87e4-4307-9066-1e6a03b62ad0-metrics-server-audit-profiles\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.534519 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-secret-metrics-server-tls\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.534646 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.534646 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-client-ca-bundle\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.534646 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.534612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93549ecb-87e4-4307-9066-1e6a03b62ad0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635604 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-client-ca-bundle\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93549ecb-87e4-4307-9066-1e6a03b62ad0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thftc\" (UniqueName: \"kubernetes.io/projected/93549ecb-87e4-4307-9066-1e6a03b62ad0-kube-api-access-thftc\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93549ecb-87e4-4307-9066-1e6a03b62ad0-audit-log\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93549ecb-87e4-4307-9066-1e6a03b62ad0-metrics-server-audit-profiles\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635919 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-secret-metrics-server-tls\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.635919 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.635911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.636164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.636142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/93549ecb-87e4-4307-9066-1e6a03b62ad0-audit-log\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.636429 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.636368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93549ecb-87e4-4307-9066-1e6a03b62ad0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.636930 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.636908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/93549ecb-87e4-4307-9066-1e6a03b62ad0-metrics-server-audit-profiles\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.638128 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.638104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-secret-metrics-server-client-certs\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.638257 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.638240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-secret-metrics-server-tls\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.638318 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.638299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/93549ecb-87e4-4307-9066-1e6a03b62ad0-client-ca-bundle\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.643427 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.643407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thftc\" (UniqueName: \"kubernetes.io/projected/93549ecb-87e4-4307-9066-1e6a03b62ad0-kube-api-access-thftc\") pod \"metrics-server-5f9c5745b-bmd45\" (UID: \"93549ecb-87e4-4307-9066-1e6a03b62ad0\") " pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.772516 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.772429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:10.788755 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.788719 2564 patch_prober.go:28] interesting pod/image-registry-78b5b6cc4c-vc4xx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 18 13:37:10.788901 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.788785 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" podUID="76a4b88f-fa98-401e-9698-74492b1da989" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 13:37:10.902246 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:10.902215 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f9c5745b-bmd45"] Jul 18 13:37:10.905137 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:37:10.905099 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93549ecb_87e4_4307_9066_1e6a03b62ad0.slice/crio-fa1369dd50d74f0f36b8a7f806a560fca06e065e5fb831fd77ced414af475bb1 WatchSource:0}: Error finding container fa1369dd50d74f0f36b8a7f806a560fca06e065e5fb831fd77ced414af475bb1: Status 404 returned error can't find the container with id fa1369dd50d74f0f36b8a7f806a560fca06e065e5fb831fd77ced414af475bb1 Jul 18 13:37:11.375304 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.375270 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55f967d76d-7749t"] Jul 18 13:37:11.379765 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.379746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.382252 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.382225 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 18 13:37:11.382369 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.382260 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 18 13:37:11.382369 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.382297 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 18 13:37:11.382569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.382547 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 18 13:37:11.382688 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.382673 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 18 13:37:11.382733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.382678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-94rpm\"" Jul 18 13:37:11.391249 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.391221 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55f967d76d-7749t"] Jul 18 13:37:11.392630 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.392588 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 18 13:37:11.444714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.444714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-telemeter-client-tls\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.444947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-federate-client-tls\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.444947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-serving-certs-ca-bundle\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.444947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.444947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444940 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvpt\" (UniqueName: \"kubernetes.io/projected/ea6d35ad-7f12-43a8-bd7b-63a434560133-kube-api-access-bxvpt\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.445097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-metrics-client-ca\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.445097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.444988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-secret-telemeter-client\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546029 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.545990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-serving-certs-ca-bundle\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvpt\" (UniqueName: \"kubernetes.io/projected/ea6d35ad-7f12-43a8-bd7b-63a434560133-kube-api-access-bxvpt\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-metrics-client-ca\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-secret-telemeter-client\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-telemeter-client-tls\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.546574 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.546531 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-federate-client-tls\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.547060 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.547027 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-metrics-client-ca\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.547228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.547190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.547228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.547160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea6d35ad-7f12-43a8-bd7b-63a434560133-serving-certs-ca-bundle\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.549480 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.549428 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-secret-telemeter-client\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.549609 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.549584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.549685 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.549672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-federate-client-tls\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.549736 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.549671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea6d35ad-7f12-43a8-bd7b-63a434560133-telemeter-client-tls\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.555206 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.555180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvpt\" (UniqueName: \"kubernetes.io/projected/ea6d35ad-7f12-43a8-bd7b-63a434560133-kube-api-access-bxvpt\") pod \"telemeter-client-55f967d76d-7749t\" (UID: \"ea6d35ad-7f12-43a8-bd7b-63a434560133\") " pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.691716 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.691688 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" Jul 18 13:37:11.845038 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.844688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55f967d76d-7749t"] Jul 18 13:37:11.847734 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:37:11.847701 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6d35ad_7f12_43a8_bd7b_63a434560133.slice/crio-0e89af823ac84f19e435a765641a0b8b30e5044196632b0138ca9dac044cd3e6 WatchSource:0}: Error finding container 0e89af823ac84f19e435a765641a0b8b30e5044196632b0138ca9dac044cd3e6: Status 404 returned error can't find the container with id 0e89af823ac84f19e435a765641a0b8b30e5044196632b0138ca9dac044cd3e6 Jul 18 13:37:11.878439 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.878399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" event={"ID":"ea6d35ad-7f12-43a8-bd7b-63a434560133","Type":"ContainerStarted","Data":"0e89af823ac84f19e435a765641a0b8b30e5044196632b0138ca9dac044cd3e6"} Jul 18 13:37:11.879840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:11.879809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" event={"ID":"93549ecb-87e4-4307-9066-1e6a03b62ad0","Type":"ContainerStarted","Data":"fa1369dd50d74f0f36b8a7f806a560fca06e065e5fb831fd77ced414af475bb1"} Jul 18 13:37:12.474689 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:37:12.474644 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-k9pcn" podUID="2e9d48b8-6612-4efd-9c5c-34bbaa70457e" Jul 18 13:37:12.885739 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.885698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" event={"ID":"93549ecb-87e4-4307-9066-1e6a03b62ad0","Type":"ContainerStarted","Data":"c32ad5a9ef72f83d358847635f5645ef408164ce1272de5657ac8c1c40344317"} Jul 18 13:37:12.885909 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.885716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9pcn" Jul 18 13:37:12.891029 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.891001 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:12.891197 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.891049 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:12.892569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.892543 2564 patch_prober.go:28] interesting pod/console-75f88c6797-dmk7f container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" start-of-body= Jul 18 13:37:12.892682 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.892603 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-75f88c6797-dmk7f" podUID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" containerName="console" probeResult="failure" output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" Jul 18 13:37:12.901502 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:12.901452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" podStartSLOduration=1.423458272 podStartE2EDuration="2.901436306s" totalImagesPullingTime="1.477978034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:12.90100443 +0000 UTC m=+283.450569112" watchObservedRunningTime="2026-07-18 13:37:12.901436306 +0000 UTC m=+283.451000988" Jul 18 13:37:13.890550 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:13.890517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" event={"ID":"ea6d35ad-7f12-43a8-bd7b-63a434560133","Type":"ContainerStarted","Data":"b4d92a22e44663aaed4df68426f7d95d4a989b419933fc30db34e70a7d59d82d"} Jul 18 13:37:14.895949 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:14.895919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" event={"ID":"ea6d35ad-7f12-43a8-bd7b-63a434560133","Type":"ContainerStarted","Data":"409104b2eef6525b12ef109141fa18cb209a29581e2d8364d5a5bddcbb9eda99"} Jul 18 13:37:15.781880 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:15.781832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:37:15.784143 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:15.784122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9d48b8-6612-4efd-9c5c-34bbaa70457e-metrics-tls\") pod \"dns-default-k9pcn\" (UID: \"2e9d48b8-6612-4efd-9c5c-34bbaa70457e\") " pod="openshift-dns/dns-default-k9pcn" Jul 18 13:37:15.888916 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:15.888877 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t94bf\"" Jul 18 13:37:15.896485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:15.896461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9pcn" Jul 18 13:37:15.901056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:15.901032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" event={"ID":"ea6d35ad-7f12-43a8-bd7b-63a434560133","Type":"ContainerStarted","Data":"79805c2213e12259019d20a2cd4091318a79d642ef3bd3deef8260a75bb17309"} Jul 18 13:37:15.924324 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:15.924264 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55f967d76d-7749t" podStartSLOduration=2.051174881 podStartE2EDuration="4.924243925s" totalImagesPullingTime="2.873069044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:37:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:15.922591028 +0000 UTC m=+286.472155708" watchObservedRunningTime="2026-07-18 13:37:15.924243925 +0000 UTC m=+286.473808607" Jul 18 13:37:16.022760 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:16.022602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9pcn"] Jul 18 13:37:16.025277 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:37:16.025253 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9d48b8_6612_4efd_9c5c_34bbaa70457e.slice/crio-3d48e690be674b9c09cba8b8b3a6aa359bb851b494cec3700d76766a46f9d229 WatchSource:0}: Error finding container 3d48e690be674b9c09cba8b8b3a6aa359bb851b494cec3700d76766a46f9d229: Status 404 returned error can't find the container with id 3d48e690be674b9c09cba8b8b3a6aa359bb851b494cec3700d76766a46f9d229 Jul 18 13:37:16.674494 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:16.674458 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f88c6797-dmk7f"] Jul 18 13:37:16.906450 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:16.906408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9pcn" event={"ID":"2e9d48b8-6612-4efd-9c5c-34bbaa70457e","Type":"ContainerStarted","Data":"3d48e690be674b9c09cba8b8b3a6aa359bb851b494cec3700d76766a46f9d229"} Jul 18 13:37:17.912271 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:17.912208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9pcn" event={"ID":"2e9d48b8-6612-4efd-9c5c-34bbaa70457e","Type":"ContainerStarted","Data":"dba254d5e04b51ab47ae01f9a92a824280d74e7030ca8a733c7278a44af7b34c"} Jul 18 13:37:18.623829 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:18.623791 2564 patch_prober.go:28] interesting pod/image-registry-78b5b6cc4c-vc4xx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 18 13:37:18.623989 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:18.623845 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" podUID="76a4b88f-fa98-401e-9698-74492b1da989" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 13:37:18.917342 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:18.917258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9pcn" event={"ID":"2e9d48b8-6612-4efd-9c5c-34bbaa70457e","Type":"ContainerStarted","Data":"aef20d23d34f65c8d93350ba1d010a5322236e9bbc76284f34d0fc4707f9945d"} Jul 18 13:37:18.917700 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:18.917409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k9pcn" Jul 18 13:37:18.934123 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:18.934076 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k9pcn" podStartSLOduration=252.164825676 podStartE2EDuration="4m13.934062286s" totalImagesPullingTime="1.76923661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:37:18.933309608 +0000 UTC m=+289.482874286" watchObservedRunningTime="2026-07-18 13:37:18.934062286 +0000 UTC m=+289.483626966" Jul 18 13:37:20.787944 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:20.787917 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78b5b6cc4c-vc4xx" Jul 18 13:37:25.850697 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:25.850623 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" podUID="f0160101-a0b4-4a99-bffc-833782c0cb2d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 18 13:37:28.923713 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:28.923673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k9pcn" Jul 18 13:37:29.850099 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:29.850076 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 18 13:37:30.773148 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:30.773117 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:30.774867 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:30.773171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:35.850700 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:35.850660 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" podUID="f0160101-a0b4-4a99-bffc-833782c0cb2d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 18 13:37:40.994311 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:40.994278 2564 generic.go:356] "Generic (PLEG): container finished" podID="2f1d947a-076d-472b-ad74-073668f6f035" containerID="83f7b73296db04ef5ee54c637eebf9b0578a26cb89597974f7fe5c5c9d37c1bb" exitCode=0 Jul 18 13:37:40.994714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:40.994336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" event={"ID":"2f1d947a-076d-472b-ad74-073668f6f035","Type":"ContainerDied","Data":"83f7b73296db04ef5ee54c637eebf9b0578a26cb89597974f7fe5c5c9d37c1bb"} Jul 18 13:37:40.994714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:40.994640 2564 scope.go:117] "RemoveContainer" containerID="83f7b73296db04ef5ee54c637eebf9b0578a26cb89597974f7fe5c5c9d37c1bb" Jul 18 13:37:41.695633 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.695571 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75f88c6797-dmk7f" podUID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" containerName="console" containerID="cri-o://2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc" gracePeriod=15 Jul 18 13:37:41.940285 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.940257 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:41.980650 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-service-ca\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.980650 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-serving-cert\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.980650 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-trusted-ca-bundle\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.980916 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980672 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-config\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.980916 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980691 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-oauth-config\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.980916 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2hs7\" (UniqueName: \"kubernetes.io/projected/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-kube-api-access-s2hs7\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.980916 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.980726 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-oauth-serving-cert\") pod \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\" (UID: \"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0\") " Jul 18 13:37:41.981124 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.981053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-service-ca" (OuterVolumeSpecName: "service-ca") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:37:41.981252 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.981224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:37:41.981252 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.981234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-config" (OuterVolumeSpecName: "console-config") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:37:41.981456 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.981289 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:37:41.982972 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.982948 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:37:41.983056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.983002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:37:41.983056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:41.983029 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-kube-api-access-s2hs7" (OuterVolumeSpecName: "kube-api-access-s2hs7") pod "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" (UID: "667bb59a-aa47-4dc7-8060-4e3f24ce4ec0"). InnerVolumeSpecName "kube-api-access-s2hs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:37:42.000302 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.000269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x6cl4" event={"ID":"2f1d947a-076d-472b-ad74-073668f6f035","Type":"ContainerStarted","Data":"6faf99d93677769db492b0a5bc0125634e2863dccfc773072e2660b508ff6b3f"} Jul 18 13:37:42.001605 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.001584 2564 generic.go:356] "Generic (PLEG): container finished" podID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" containerID="2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc" exitCode=0 Jul 18 13:37:42.001717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.001612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f88c6797-dmk7f" event={"ID":"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0","Type":"ContainerDied","Data":"2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc"} Jul 18 13:37:42.001717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.001613 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f88c6797-dmk7f" Jul 18 13:37:42.001717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.001647 2564 scope.go:117] "RemoveContainer" containerID="2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc" Jul 18 13:37:42.001866 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.001638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f88c6797-dmk7f" event={"ID":"667bb59a-aa47-4dc7-8060-4e3f24ce4ec0","Type":"ContainerDied","Data":"a53a35ec63690331536f0fb217e0f5fcfd06286b392883a8f05be0374186ed0a"} Jul 18 13:37:42.013892 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.013868 2564 scope.go:117] "RemoveContainer" containerID="2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc" Jul 18 13:37:42.014133 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:37:42.014114 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc\": container with ID starting with 2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc not found: ID does not exist" containerID="2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc" Jul 18 13:37:42.014190 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.014141 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc"} err="failed to get container status \"2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc\": rpc error: code = NotFound desc = could not find container \"2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc\": container with ID starting with 2097cd3fc584c6c8285c853e017678681be6e5790eb8cc4b1ccafce0cbc8b5bc not found: ID does not exist" Jul 18 13:37:42.038422 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.038367 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f88c6797-dmk7f"] Jul 18 13:37:42.044917 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.044889 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f88c6797-dmk7f"] Jul 18 13:37:42.081352 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081317 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:42.081352 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081342 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-serving-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:42.081352 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081351 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-trusted-ca-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:42.081352 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081360 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:42.081675 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081369 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-console-oauth-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:42.081675 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081394 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2hs7\" (UniqueName: \"kubernetes.io/projected/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-kube-api-access-s2hs7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:42.081675 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:42.081403 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0-oauth-serving-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:37:43.971427 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:43.971394 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" path="/var/lib/kubelet/pods/667bb59a-aa47-4dc7-8060-4e3f24ce4ec0/volumes" Jul 18 13:37:45.850062 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:45.850019 2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" podUID="f0160101-a0b4-4a99-bffc-833782c0cb2d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 18 13:37:45.850459 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:45.850109 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" Jul 18 13:37:45.850618 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:45.850600 2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b74b4cd8e730e1cb92ad0c2d6c2c8ddb69288aff16599d2f944f3443b48f9a05"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 18 13:37:45.850661 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:45.850642 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" podUID="f0160101-a0b4-4a99-bffc-833782c0cb2d" containerName="service-proxy" containerID="cri-o://b74b4cd8e730e1cb92ad0c2d6c2c8ddb69288aff16599d2f944f3443b48f9a05" gracePeriod=30 Jul 18 13:37:46.020145 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:46.020117 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0160101-a0b4-4a99-bffc-833782c0cb2d" containerID="b74b4cd8e730e1cb92ad0c2d6c2c8ddb69288aff16599d2f944f3443b48f9a05" exitCode=2 Jul 18 13:37:46.020252 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:46.020179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" event={"ID":"f0160101-a0b4-4a99-bffc-833782c0cb2d","Type":"ContainerDied","Data":"b74b4cd8e730e1cb92ad0c2d6c2c8ddb69288aff16599d2f944f3443b48f9a05"} Jul 18 13:37:47.026236 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:47.026201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b6f765bd-lpcss" event={"ID":"f0160101-a0b4-4a99-bffc-833782c0cb2d","Type":"ContainerStarted","Data":"816ca4a9326b7775c514f3cf7ec068117603cf60f498b3541af02d7fa9cc4f0e"} Jul 18 13:37:50.778578 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:50.778545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:37:50.782570 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:37:50.782548 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f9c5745b-bmd45" Jul 18 13:38:49.477418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.477360 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7847b7df5-7fp9r"] Jul 18 13:38:49.477845 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.477754 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" containerName="console" Jul 18 13:38:49.477845 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.477770 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" containerName="console" Jul 18 13:38:49.477845 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.477829 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="667bb59a-aa47-4dc7-8060-4e3f24ce4ec0" containerName="console" Jul 18 13:38:49.481819 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.481803 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.484303 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.484272 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 18 13:38:49.484450 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.484414 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 18 13:38:49.484450 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.484426 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 18 13:38:49.485573 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.485553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 18 13:38:49.486107 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.486082 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 18 13:38:49.486210 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.486185 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 18 13:38:49.486273 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.486215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh5bg\"" Jul 18 13:38:49.486326 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.486282 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 18 13:38:49.491148 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.491123 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7847b7df5-7fp9r"] Jul 18 13:38:49.494354 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.494331 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 18 13:38:49.604541 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-oauth-config\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.604711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-config\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.604711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-oauth-serving-cert\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.604711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-serving-cert\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.604711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-service-ca\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.604711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgk6n\" (UniqueName: \"kubernetes.io/projected/7308cc43-8acb-407b-92b2-0a4d3f98df52-kube-api-access-mgk6n\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.604711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.604678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-trusted-ca-bundle\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705534 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-serving-cert\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-service-ca\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgk6n\" (UniqueName: \"kubernetes.io/projected/7308cc43-8acb-407b-92b2-0a4d3f98df52-kube-api-access-mgk6n\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-trusted-ca-bundle\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-oauth-config\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-config\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.705711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.705700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-oauth-serving-cert\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.706474 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.706438 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-config\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.706577 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.706443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-oauth-serving-cert\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.706577 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.706444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-service-ca\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.706577 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.706556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-trusted-ca-bundle\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.708151 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.708129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-serving-cert\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.708233 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.708156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-oauth-config\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.713526 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.713505 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgk6n\" (UniqueName: \"kubernetes.io/projected/7308cc43-8acb-407b-92b2-0a4d3f98df52-kube-api-access-mgk6n\") pod \"console-7847b7df5-7fp9r\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.794773 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.794690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:49.935595 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:49.935562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7847b7df5-7fp9r"] Jul 18 13:38:49.938317 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:38:49.938289 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7308cc43_8acb_407b_92b2_0a4d3f98df52.slice/crio-30b7994478cda83ba21c5bf2a49976ecff4f502602236d877165d60a1d7705f6 WatchSource:0}: Error finding container 30b7994478cda83ba21c5bf2a49976ecff4f502602236d877165d60a1d7705f6: Status 404 returned error can't find the container with id 30b7994478cda83ba21c5bf2a49976ecff4f502602236d877165d60a1d7705f6 Jul 18 13:38:50.252806 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:50.252770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847b7df5-7fp9r" event={"ID":"7308cc43-8acb-407b-92b2-0a4d3f98df52","Type":"ContainerStarted","Data":"842f8047dc9a66a495abac96f844ef1c2e15cddd666e2dbb78a918139a1873a5"} Jul 18 13:38:50.253019 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:50.252817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847b7df5-7fp9r" event={"ID":"7308cc43-8acb-407b-92b2-0a4d3f98df52","Type":"ContainerStarted","Data":"30b7994478cda83ba21c5bf2a49976ecff4f502602236d877165d60a1d7705f6"} Jul 18 13:38:50.269343 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:50.269291 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7847b7df5-7fp9r" podStartSLOduration=1.269273971 podStartE2EDuration="1.269273971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:38:50.268507184 +0000 UTC m=+380.818071865" watchObservedRunningTime="2026-07-18 13:38:50.269273971 +0000 UTC m=+380.818838652" Jul 18 13:38:59.795211 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:59.795167 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:59.795650 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:59.795260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:38:59.800060 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:38:59.800037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:39:00.292643 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:00.292617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:39:29.468326 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.468279 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn"] Jul 18 13:39:29.471454 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.471434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.473952 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.473923 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kfclk\"" Jul 18 13:39:29.474110 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.474092 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 18 13:39:29.475147 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.475130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 18 13:39:29.481783 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.481737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn"] Jul 18 13:39:29.610269 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.610233 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.610269 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.610276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9g6\" (UniqueName: \"kubernetes.io/projected/cffad89e-8287-41fc-a519-374a83370336-kube-api-access-8w9g6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.610524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.610394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.711430 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.711395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.711430 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.711436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9g6\" (UniqueName: \"kubernetes.io/projected/cffad89e-8287-41fc-a519-374a83370336-kube-api-access-8w9g6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.711689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.711491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.711866 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.711849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.711901 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.711848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.719840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.719772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9g6\" (UniqueName: \"kubernetes.io/projected/cffad89e-8287-41fc-a519-374a83370336-kube-api-access-8w9g6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.781004 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.780972 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:29.910548 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.910523 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn"] Jul 18 13:39:29.913017 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:39:29.912983 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffad89e_8287_41fc_a519_374a83370336.slice/crio-eb2c2940ab885f56eeb00bb6051f102440ab9f6774e52a5ad9ce0e6038be40f4 WatchSource:0}: Error finding container eb2c2940ab885f56eeb00bb6051f102440ab9f6774e52a5ad9ce0e6038be40f4: Status 404 returned error can't find the container with id eb2c2940ab885f56eeb00bb6051f102440ab9f6774e52a5ad9ce0e6038be40f4 Jul 18 13:39:29.915359 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:29.915346 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 13:39:30.393644 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:30.393615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" event={"ID":"cffad89e-8287-41fc-a519-374a83370336","Type":"ContainerStarted","Data":"eb2c2940ab885f56eeb00bb6051f102440ab9f6774e52a5ad9ce0e6038be40f4"} Jul 18 13:39:36.417809 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:36.417769 2564 generic.go:356] "Generic (PLEG): container finished" podID="cffad89e-8287-41fc-a519-374a83370336" containerID="1939369248f6acc71983ac0120826140b2134085a336fd02e881b70349ca4ce6" exitCode=0 Jul 18 13:39:36.418221 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:36.417858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" event={"ID":"cffad89e-8287-41fc-a519-374a83370336","Type":"ContainerDied","Data":"1939369248f6acc71983ac0120826140b2134085a336fd02e881b70349ca4ce6"} Jul 18 13:39:38.428213 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:38.428182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" event={"ID":"cffad89e-8287-41fc-a519-374a83370336","Type":"ContainerStarted","Data":"3a1231935e645246cdad6d586be0fcf1de51bcb146897929e06e6a869fff0c5e"} Jul 18 13:39:39.434080 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:39.434044 2564 generic.go:356] "Generic (PLEG): container finished" podID="cffad89e-8287-41fc-a519-374a83370336" containerID="3a1231935e645246cdad6d586be0fcf1de51bcb146897929e06e6a869fff0c5e" exitCode=0 Jul 18 13:39:39.434470 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:39.434087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" event={"ID":"cffad89e-8287-41fc-a519-374a83370336","Type":"ContainerDied","Data":"3a1231935e645246cdad6d586be0fcf1de51bcb146897929e06e6a869fff0c5e"} Jul 18 13:39:46.471252 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:46.471212 2564 generic.go:356] "Generic (PLEG): container finished" podID="cffad89e-8287-41fc-a519-374a83370336" containerID="ef87e9b29d07b95b10428cdbbf3fd2949860d4560aa2c827d78946687ab4e62c" exitCode=0 Jul 18 13:39:46.471679 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:46.471296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" event={"ID":"cffad89e-8287-41fc-a519-374a83370336","Type":"ContainerDied","Data":"ef87e9b29d07b95b10428cdbbf3fd2949860d4560aa2c827d78946687ab4e62c"} Jul 18 13:39:47.603227 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.603201 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:47.771621 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.771523 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w9g6\" (UniqueName: \"kubernetes.io/projected/cffad89e-8287-41fc-a519-374a83370336-kube-api-access-8w9g6\") pod \"cffad89e-8287-41fc-a519-374a83370336\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " Jul 18 13:39:47.771621 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.771578 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-bundle\") pod \"cffad89e-8287-41fc-a519-374a83370336\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " Jul 18 13:39:47.771840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.771638 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-util\") pod \"cffad89e-8287-41fc-a519-374a83370336\" (UID: \"cffad89e-8287-41fc-a519-374a83370336\") " Jul 18 13:39:47.772225 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.772193 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-bundle" (OuterVolumeSpecName: "bundle") pod "cffad89e-8287-41fc-a519-374a83370336" (UID: "cffad89e-8287-41fc-a519-374a83370336"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:39:47.773745 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.773711 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffad89e-8287-41fc-a519-374a83370336-kube-api-access-8w9g6" (OuterVolumeSpecName: "kube-api-access-8w9g6") pod "cffad89e-8287-41fc-a519-374a83370336" (UID: "cffad89e-8287-41fc-a519-374a83370336"). InnerVolumeSpecName "kube-api-access-8w9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:39:47.776617 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.776589 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-util" (OuterVolumeSpecName: "util") pod "cffad89e-8287-41fc-a519-374a83370336" (UID: "cffad89e-8287-41fc-a519-374a83370336"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:39:47.872477 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.872427 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9g6\" (UniqueName: \"kubernetes.io/projected/cffad89e-8287-41fc-a519-374a83370336-kube-api-access-8w9g6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:39:47.872477 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.872472 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:39:47.872477 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:47.872481 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cffad89e-8287-41fc-a519-374a83370336-util\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:39:48.479872 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:48.479845 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" Jul 18 13:39:48.479872 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:48.479868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540fpxqn" event={"ID":"cffad89e-8287-41fc-a519-374a83370336","Type":"ContainerDied","Data":"eb2c2940ab885f56eeb00bb6051f102440ab9f6774e52a5ad9ce0e6038be40f4"} Jul 18 13:39:48.480077 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:48.479910 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2c2940ab885f56eeb00bb6051f102440ab9f6774e52a5ad9ce0e6038be40f4" Jul 18 13:39:52.442336 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442263 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp"] Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442540 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="pull" Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442551 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="pull" Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442567 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="util" Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442572 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="util" Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442582 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="extract" Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442587 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="extract" Jul 18 13:39:52.442707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.442629 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffad89e-8287-41fc-a519-374a83370336" containerName="extract" Jul 18 13:39:52.457856 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.457832 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.460957 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.460932 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 18 13:39:52.461221 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.461201 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 18 13:39:52.461791 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.461774 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nht5p\"" Jul 18 13:39:52.469307 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.469281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp"] Jul 18 13:39:52.611117 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.611082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf6zq\" (UniqueName: \"kubernetes.io/projected/1761315a-0ac3-4f96-9a85-408008ba8801-kube-api-access-gf6zq\") pod \"cert-manager-operator-controller-manager-55c444bf86-5cctp\" (UID: \"1761315a-0ac3-4f96-9a85-408008ba8801\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.611281 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.611151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1761315a-0ac3-4f96-9a85-408008ba8801-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5cctp\" (UID: \"1761315a-0ac3-4f96-9a85-408008ba8801\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.711757 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.711667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1761315a-0ac3-4f96-9a85-408008ba8801-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5cctp\" (UID: \"1761315a-0ac3-4f96-9a85-408008ba8801\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.711757 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.711727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf6zq\" (UniqueName: \"kubernetes.io/projected/1761315a-0ac3-4f96-9a85-408008ba8801-kube-api-access-gf6zq\") pod \"cert-manager-operator-controller-manager-55c444bf86-5cctp\" (UID: \"1761315a-0ac3-4f96-9a85-408008ba8801\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.712052 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.712031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1761315a-0ac3-4f96-9a85-408008ba8801-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5cctp\" (UID: \"1761315a-0ac3-4f96-9a85-408008ba8801\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.722166 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.722141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf6zq\" (UniqueName: \"kubernetes.io/projected/1761315a-0ac3-4f96-9a85-408008ba8801-kube-api-access-gf6zq\") pod \"cert-manager-operator-controller-manager-55c444bf86-5cctp\" (UID: \"1761315a-0ac3-4f96-9a85-408008ba8801\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.767262 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.767225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" Jul 18 13:39:52.913127 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:52.913096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp"] Jul 18 13:39:52.917029 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:39:52.917005 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1761315a_0ac3_4f96_9a85_408008ba8801.slice/crio-0bd269178b3048677bc91cbd17de2fe8e1ef54adf96909d988970edfeebd3e1b WatchSource:0}: Error finding container 0bd269178b3048677bc91cbd17de2fe8e1ef54adf96909d988970edfeebd3e1b: Status 404 returned error can't find the container with id 0bd269178b3048677bc91cbd17de2fe8e1ef54adf96909d988970edfeebd3e1b Jul 18 13:39:53.502234 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:53.502192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" event={"ID":"1761315a-0ac3-4f96-9a85-408008ba8801","Type":"ContainerStarted","Data":"0bd269178b3048677bc91cbd17de2fe8e1ef54adf96909d988970edfeebd3e1b"} Jul 18 13:39:55.510842 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:55.510811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" event={"ID":"1761315a-0ac3-4f96-9a85-408008ba8801","Type":"ContainerStarted","Data":"d9439c6d8bb94eda5f6192ba1ffc73d9edcdc72cd3a1f503db61b3b42e5a716d"} Jul 18 13:39:55.533732 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:55.533671 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5cctp" podStartSLOduration=1.6346903959999999 podStartE2EDuration="3.533649198s" totalImagesPullingTime="1.898958802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:39:55.532293379 +0000 UTC m=+446.081858077" watchObservedRunningTime="2026-07-18 13:39:55.533649198 +0000 UTC m=+446.083213880" Jul 18 13:39:58.859386 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.859337 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs"] Jul 18 13:39:58.861403 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.861364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:58.863840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.863815 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 18 13:39:58.864907 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.864880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2tf7c\"" Jul 18 13:39:58.865013 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.864919 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 18 13:39:58.872104 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.872082 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs"] Jul 18 13:39:58.961832 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.961794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da183d32-92ea-4100-a6dd-745352976e4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ncqcs\" (UID: \"da183d32-92ea-4100-a6dd-745352976e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:58.962009 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:58.961925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbgg\" (UniqueName: \"kubernetes.io/projected/da183d32-92ea-4100-a6dd-745352976e4a-kube-api-access-mkbgg\") pod \"cert-manager-cainjector-559cdd8c87-ncqcs\" (UID: \"da183d32-92ea-4100-a6dd-745352976e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:59.062641 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.062607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbgg\" (UniqueName: \"kubernetes.io/projected/da183d32-92ea-4100-a6dd-745352976e4a-kube-api-access-mkbgg\") pod \"cert-manager-cainjector-559cdd8c87-ncqcs\" (UID: \"da183d32-92ea-4100-a6dd-745352976e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:59.062785 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.062655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da183d32-92ea-4100-a6dd-745352976e4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ncqcs\" (UID: \"da183d32-92ea-4100-a6dd-745352976e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:59.073103 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.073066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da183d32-92ea-4100-a6dd-745352976e4a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ncqcs\" (UID: \"da183d32-92ea-4100-a6dd-745352976e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:59.073228 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.073181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbgg\" (UniqueName: \"kubernetes.io/projected/da183d32-92ea-4100-a6dd-745352976e4a-kube-api-access-mkbgg\") pod \"cert-manager-cainjector-559cdd8c87-ncqcs\" (UID: \"da183d32-92ea-4100-a6dd-745352976e4a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:59.170244 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.170149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" Jul 18 13:39:59.296526 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.296495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs"] Jul 18 13:39:59.299348 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:39:59.299320 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda183d32_92ea_4100_a6dd_745352976e4a.slice/crio-c0140870764d9fe45dd78037e4d3a4fd97e140447c47c93eb53db1caf9a08329 WatchSource:0}: Error finding container c0140870764d9fe45dd78037e4d3a4fd97e140447c47c93eb53db1caf9a08329: Status 404 returned error can't find the container with id c0140870764d9fe45dd78037e4d3a4fd97e140447c47c93eb53db1caf9a08329 Jul 18 13:39:59.528719 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:39:59.528683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" event={"ID":"da183d32-92ea-4100-a6dd-745352976e4a","Type":"ContainerStarted","Data":"c0140870764d9fe45dd78037e4d3a4fd97e140447c47c93eb53db1caf9a08329"} Jul 18 13:40:02.542232 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:02.542192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" event={"ID":"da183d32-92ea-4100-a6dd-745352976e4a","Type":"ContainerStarted","Data":"3f8725ca860484446920c397b0282f04380e66ef41dcfc5987c843ddcda05db2"} Jul 18 13:40:02.558222 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:02.558173 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ncqcs" podStartSLOduration=2.296557716 podStartE2EDuration="4.558154585s" totalImagesPullingTime="2.261596869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:40:02.557002767 +0000 UTC m=+453.106567450" watchObservedRunningTime="2026-07-18 13:40:02.558154585 +0000 UTC m=+453.107719266" Jul 18 13:40:20.736118 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.736075 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5"] Jul 18 13:40:20.738508 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.738491 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:20.741016 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.740974 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 18 13:40:20.741159 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.741034 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 18 13:40:20.741159 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.741071 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pvkks\"" Jul 18 13:40:20.749422 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.749340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5"] Jul 18 13:40:20.834863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.834824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzph\" (UniqueName: \"kubernetes.io/projected/a322fc9d-ddac-4d81-a206-66f305c66a45-kube-api-access-qqzph\") pod \"openshift-lws-operator-fd8ccff4c-lvkr5\" (UID: \"a322fc9d-ddac-4d81-a206-66f305c66a45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:20.834863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.834876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a322fc9d-ddac-4d81-a206-66f305c66a45-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lvkr5\" (UID: \"a322fc9d-ddac-4d81-a206-66f305c66a45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:20.935910 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.935865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzph\" (UniqueName: \"kubernetes.io/projected/a322fc9d-ddac-4d81-a206-66f305c66a45-kube-api-access-qqzph\") pod \"openshift-lws-operator-fd8ccff4c-lvkr5\" (UID: \"a322fc9d-ddac-4d81-a206-66f305c66a45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:20.936061 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.935921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a322fc9d-ddac-4d81-a206-66f305c66a45-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lvkr5\" (UID: \"a322fc9d-ddac-4d81-a206-66f305c66a45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:20.936334 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.936314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a322fc9d-ddac-4d81-a206-66f305c66a45-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lvkr5\" (UID: \"a322fc9d-ddac-4d81-a206-66f305c66a45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:20.944453 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:20.944416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzph\" (UniqueName: \"kubernetes.io/projected/a322fc9d-ddac-4d81-a206-66f305c66a45-kube-api-access-qqzph\") pod \"openshift-lws-operator-fd8ccff4c-lvkr5\" (UID: \"a322fc9d-ddac-4d81-a206-66f305c66a45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:21.046947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:21.046859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" Jul 18 13:40:21.177844 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:21.177811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5"] Jul 18 13:40:21.180133 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:40:21.180105 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda322fc9d_ddac_4d81_a206_66f305c66a45.slice/crio-e4028c177aaea2957da7fb008b9d9fa13a9c7f3609592c21ff1d86a33123bdda WatchSource:0}: Error finding container e4028c177aaea2957da7fb008b9d9fa13a9c7f3609592c21ff1d86a33123bdda: Status 404 returned error can't find the container with id e4028c177aaea2957da7fb008b9d9fa13a9c7f3609592c21ff1d86a33123bdda Jul 18 13:40:21.616421 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:21.616362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" event={"ID":"a322fc9d-ddac-4d81-a206-66f305c66a45","Type":"ContainerStarted","Data":"e4028c177aaea2957da7fb008b9d9fa13a9c7f3609592c21ff1d86a33123bdda"} Jul 18 13:40:24.632164 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:24.632127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" event={"ID":"a322fc9d-ddac-4d81-a206-66f305c66a45","Type":"ContainerStarted","Data":"dc95655ef575a5ec167a1b53901b731bb44a741bb440ccba8979308bca86109d"} Jul 18 13:40:24.649073 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:24.649004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lvkr5" podStartSLOduration=2.099929928 podStartE2EDuration="4.648986153s" totalImagesPullingTime="2.549056225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:40:24.647064578 +0000 UTC m=+475.196629274" watchObservedRunningTime="2026-07-18 13:40:24.648986153 +0000 UTC m=+475.198550837" Jul 18 13:40:28.353007 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.352968 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2"] Jul 18 13:40:28.357193 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.357175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.360105 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.360076 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 18 13:40:28.360236 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.360216 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 18 13:40:28.361234 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.361215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kfclk\"" Jul 18 13:40:28.365530 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.365508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2"] Jul 18 13:40:28.401196 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.401158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.401196 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.401200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzk2\" (UniqueName: \"kubernetes.io/projected/8516b7b8-2e0b-4753-bca1-186b1dbb653f-kube-api-access-mrzk2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.401449 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.401222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.501708 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.501672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.501708 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.501710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzk2\" (UniqueName: \"kubernetes.io/projected/8516b7b8-2e0b-4753-bca1-186b1dbb653f-kube-api-access-mrzk2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.501908 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.501734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.502073 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.502053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.502109 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.502074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.509338 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.509305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzk2\" (UniqueName: \"kubernetes.io/projected/8516b7b8-2e0b-4753-bca1-186b1dbb653f-kube-api-access-mrzk2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.667594 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.667507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:28.799034 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:28.799005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2"] Jul 18 13:40:28.801245 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:40:28.801216 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8516b7b8_2e0b_4753_bca1_186b1dbb653f.slice/crio-ada63bd8bb4c34f5bb6daa63a09f557fafccb03be0cfbb5613fc4286dad236aa WatchSource:0}: Error finding container ada63bd8bb4c34f5bb6daa63a09f557fafccb03be0cfbb5613fc4286dad236aa: Status 404 returned error can't find the container with id ada63bd8bb4c34f5bb6daa63a09f557fafccb03be0cfbb5613fc4286dad236aa Jul 18 13:40:29.654390 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:29.654350 2564 generic.go:356] "Generic (PLEG): container finished" podID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerID="2f737323b7d4ccc0e6ecaf1f65ee9e6d563356cc1eb6731b6176aa3ad268e561" exitCode=0 Jul 18 13:40:29.654855 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:29.654411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" event={"ID":"8516b7b8-2e0b-4753-bca1-186b1dbb653f","Type":"ContainerDied","Data":"2f737323b7d4ccc0e6ecaf1f65ee9e6d563356cc1eb6731b6176aa3ad268e561"} Jul 18 13:40:29.654855 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:29.654449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" event={"ID":"8516b7b8-2e0b-4753-bca1-186b1dbb653f","Type":"ContainerStarted","Data":"ada63bd8bb4c34f5bb6daa63a09f557fafccb03be0cfbb5613fc4286dad236aa"} Jul 18 13:40:30.660495 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:30.660367 2564 generic.go:356] "Generic (PLEG): container finished" podID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerID="df0437547f632fce8569bf3a2fd7ca0ee1df8f71e00db65b533c01a64a6c52c6" exitCode=0 Jul 18 13:40:30.660495 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:30.660484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" event={"ID":"8516b7b8-2e0b-4753-bca1-186b1dbb653f","Type":"ContainerDied","Data":"df0437547f632fce8569bf3a2fd7ca0ee1df8f71e00db65b533c01a64a6c52c6"} Jul 18 13:40:31.666333 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:31.666300 2564 generic.go:356] "Generic (PLEG): container finished" podID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerID="3f2805e7c041939a1396dc21088d3bee028e49c97f9260146f4e3a440a895395" exitCode=0 Jul 18 13:40:31.666749 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:31.666399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" event={"ID":"8516b7b8-2e0b-4753-bca1-186b1dbb653f","Type":"ContainerDied","Data":"3f2805e7c041939a1396dc21088d3bee028e49c97f9260146f4e3a440a895395"} Jul 18 13:40:32.805947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.805923 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:32.945802 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.945759 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-bundle\") pod \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " Jul 18 13:40:32.945971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.945880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-util\") pod \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " Jul 18 13:40:32.945971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.945938 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrzk2\" (UniqueName: \"kubernetes.io/projected/8516b7b8-2e0b-4753-bca1-186b1dbb653f-kube-api-access-mrzk2\") pod \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\" (UID: \"8516b7b8-2e0b-4753-bca1-186b1dbb653f\") " Jul 18 13:40:32.946524 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.946499 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-bundle" (OuterVolumeSpecName: "bundle") pod "8516b7b8-2e0b-4753-bca1-186b1dbb653f" (UID: "8516b7b8-2e0b-4753-bca1-186b1dbb653f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:40:32.947973 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.947950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8516b7b8-2e0b-4753-bca1-186b1dbb653f-kube-api-access-mrzk2" (OuterVolumeSpecName: "kube-api-access-mrzk2") pod "8516b7b8-2e0b-4753-bca1-186b1dbb653f" (UID: "8516b7b8-2e0b-4753-bca1-186b1dbb653f"). InnerVolumeSpecName "kube-api-access-mrzk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:40:32.951205 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:32.951172 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-util" (OuterVolumeSpecName: "util") pod "8516b7b8-2e0b-4753-bca1-186b1dbb653f" (UID: "8516b7b8-2e0b-4753-bca1-186b1dbb653f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:40:33.046782 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:33.046690 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-util\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:40:33.046782 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:33.046721 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrzk2\" (UniqueName: \"kubernetes.io/projected/8516b7b8-2e0b-4753-bca1-186b1dbb653f-kube-api-access-mrzk2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:40:33.046782 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:33.046731 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8516b7b8-2e0b-4753-bca1-186b1dbb653f-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:40:33.676090 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:33.676056 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" Jul 18 13:40:33.676090 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:33.676067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af97rq2" event={"ID":"8516b7b8-2e0b-4753-bca1-186b1dbb653f","Type":"ContainerDied","Data":"ada63bd8bb4c34f5bb6daa63a09f557fafccb03be0cfbb5613fc4286dad236aa"} Jul 18 13:40:33.676301 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:33.676134 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ada63bd8bb4c34f5bb6daa63a09f557fafccb03be0cfbb5613fc4286dad236aa" Jul 18 13:40:40.076655 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076620 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk"] Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076937 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="pull" Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076948 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="pull" Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076957 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="extract" Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076962 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="extract" Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076973 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="util" Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.076978 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="util" Jul 18 13:40:40.077036 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.077025 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8516b7b8-2e0b-4753-bca1-186b1dbb653f" containerName="extract" Jul 18 13:40:40.079471 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.079454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.084746 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.084716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 18 13:40:40.084865 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.084749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 18 13:40:40.084989 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.084975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zj28h\"" Jul 18 13:40:40.085115 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.085100 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 18 13:40:40.085263 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.085251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 18 13:40:40.103938 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.103901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk"] Jul 18 13:40:40.209137 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.209104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28dc\" (UniqueName: \"kubernetes.io/projected/d179a8f6-f972-46cc-83b1-409934056e12-kube-api-access-q28dc\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.209333 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.209165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d179a8f6-f972-46cc-83b1-409934056e12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.209333 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.209193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d179a8f6-f972-46cc-83b1-409934056e12-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.209333 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.209236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d179a8f6-f972-46cc-83b1-409934056e12-tmp\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.209333 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.209304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d179a8f6-f972-46cc-83b1-409934056e12-manifests\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.310449 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.310412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28dc\" (UniqueName: \"kubernetes.io/projected/d179a8f6-f972-46cc-83b1-409934056e12-kube-api-access-q28dc\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.310639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.310464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d179a8f6-f972-46cc-83b1-409934056e12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.310639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.310484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d179a8f6-f972-46cc-83b1-409934056e12-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.310639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.310513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d179a8f6-f972-46cc-83b1-409934056e12-tmp\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.310639 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.310539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d179a8f6-f972-46cc-83b1-409934056e12-manifests\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.311019 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.310989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d179a8f6-f972-46cc-83b1-409934056e12-manifests\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.312874 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.312844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d179a8f6-f972-46cc-83b1-409934056e12-tmp\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.312968 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.312920 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d179a8f6-f972-46cc-83b1-409934056e12-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.313085 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.313070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d179a8f6-f972-46cc-83b1-409934056e12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.318109 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.318088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28dc\" (UniqueName: \"kubernetes.io/projected/d179a8f6-f972-46cc-83b1-409934056e12-kube-api-access-q28dc\") pod \"opendatahub-operator-controller-manager-7bd7687b4d-shppk\" (UID: \"d179a8f6-f972-46cc-83b1-409934056e12\") " pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.390000 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.389913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:40.545162 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.545132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk"] Jul 18 13:40:40.547864 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:40:40.547834 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd179a8f6_f972_46cc_83b1_409934056e12.slice/crio-ecba61177cb11312eb5d7aa1dc01ae250806ac3f3e22711b4d6b7223ec4d6de2 WatchSource:0}: Error finding container ecba61177cb11312eb5d7aa1dc01ae250806ac3f3e22711b4d6b7223ec4d6de2: Status 404 returned error can't find the container with id ecba61177cb11312eb5d7aa1dc01ae250806ac3f3e22711b4d6b7223ec4d6de2 Jul 18 13:40:40.707369 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:40.707334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" event={"ID":"d179a8f6-f972-46cc-83b1-409934056e12","Type":"ContainerStarted","Data":"ecba61177cb11312eb5d7aa1dc01ae250806ac3f3e22711b4d6b7223ec4d6de2"} Jul 18 13:40:43.724857 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:43.724820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" event={"ID":"d179a8f6-f972-46cc-83b1-409934056e12","Type":"ContainerStarted","Data":"2ea54a7c26c302f6d9698abf0fb184f99250e88ad41a4ff111e44beffd342cff"} Jul 18 13:40:44.729981 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:44.729949 2564 generic.go:356] "Generic (PLEG): container finished" podID="d179a8f6-f972-46cc-83b1-409934056e12" containerID="2ea54a7c26c302f6d9698abf0fb184f99250e88ad41a4ff111e44beffd342cff" exitCode=0 Jul 18 13:40:44.730401 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:44.730042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" event={"ID":"d179a8f6-f972-46cc-83b1-409934056e12","Type":"ContainerDied","Data":"2ea54a7c26c302f6d9698abf0fb184f99250e88ad41a4ff111e44beffd342cff"} Jul 18 13:40:45.736484 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:45.736445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" event={"ID":"d179a8f6-f972-46cc-83b1-409934056e12","Type":"ContainerStarted","Data":"43ef128206eb0b43e1de8da5fc12f2adc5883ee9735e9f9694ae8aecc78ab402"} Jul 18 13:40:45.736869 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:45.736581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:40:45.759580 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:45.759530 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" podStartSLOduration=1.474469881 podStartE2EDuration="5.759515423s" totalImagesPullingTime="3.209334758s" totalInitContainerRuntime="1.075710784s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:40:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:40:45.757760904 +0000 UTC m=+496.307325585" watchObservedRunningTime="2026-07-18 13:40:45.759515423 +0000 UTC m=+496.309080105" Jul 18 13:40:56.743482 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:40:56.743450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7bd7687b4d-shppk" Jul 18 13:41:04.172857 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.172817 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv"] Jul 18 13:41:04.179684 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.179659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.185452 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.185420 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 18 13:41:04.185598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.185567 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 18 13:41:04.186706 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.186638 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 18 13:41:04.186706 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.186654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 18 13:41:04.186875 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.186706 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pdp8t\"" Jul 18 13:41:04.189546 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.189525 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv"] Jul 18 13:41:04.219404 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.219350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94017353-c705-44dc-b6df-380aa7dbd780-tls-certs\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.219591 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.219435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94017353-c705-44dc-b6df-380aa7dbd780-tmp\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.219591 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.219463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5tq\" (UniqueName: \"kubernetes.io/projected/94017353-c705-44dc-b6df-380aa7dbd780-kube-api-access-2f5tq\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.317178 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.317142 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mcpjl"] Jul 18 13:41:04.320426 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.320401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.320636 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.320612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94017353-c705-44dc-b6df-380aa7dbd780-tls-certs\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.320773 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.320673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94017353-c705-44dc-b6df-380aa7dbd780-tmp\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.320773 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.320707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5tq\" (UniqueName: \"kubernetes.io/projected/94017353-c705-44dc-b6df-380aa7dbd780-kube-api-access-2f5tq\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.323311 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.323287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94017353-c705-44dc-b6df-380aa7dbd780-tmp\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.323485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.323468 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94017353-c705-44dc-b6df-380aa7dbd780-tls-certs\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.324863 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.324845 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 18 13:41:04.324973 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.324864 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-txb56\"" Jul 18 13:41:04.332968 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.332947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mcpjl"] Jul 18 13:41:04.338884 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.338861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5tq\" (UniqueName: \"kubernetes.io/projected/94017353-c705-44dc-b6df-380aa7dbd780-kube-api-access-2f5tq\") pod \"kube-auth-proxy-7d65948f8d-qvnzv\" (UID: \"94017353-c705-44dc-b6df-380aa7dbd780\") " pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.421361 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.421324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0492ceb8-72fc-43ea-b95b-45c657ff946d-cert\") pod \"odh-model-controller-7998c67dc4-mcpjl\" (UID: \"0492ceb8-72fc-43ea-b95b-45c657ff946d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.421361 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.421362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5bt4\" (UniqueName: \"kubernetes.io/projected/0492ceb8-72fc-43ea-b95b-45c657ff946d-kube-api-access-l5bt4\") pod \"odh-model-controller-7998c67dc4-mcpjl\" (UID: \"0492ceb8-72fc-43ea-b95b-45c657ff946d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.490347 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.490308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" Jul 18 13:41:04.522611 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.522572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0492ceb8-72fc-43ea-b95b-45c657ff946d-cert\") pod \"odh-model-controller-7998c67dc4-mcpjl\" (UID: \"0492ceb8-72fc-43ea-b95b-45c657ff946d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.522611 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.522615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5bt4\" (UniqueName: \"kubernetes.io/projected/0492ceb8-72fc-43ea-b95b-45c657ff946d-kube-api-access-l5bt4\") pod \"odh-model-controller-7998c67dc4-mcpjl\" (UID: \"0492ceb8-72fc-43ea-b95b-45c657ff946d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.525166 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.525134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0492ceb8-72fc-43ea-b95b-45c657ff946d-cert\") pod \"odh-model-controller-7998c67dc4-mcpjl\" (UID: \"0492ceb8-72fc-43ea-b95b-45c657ff946d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.530885 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.530858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5bt4\" (UniqueName: \"kubernetes.io/projected/0492ceb8-72fc-43ea-b95b-45c657ff946d-kube-api-access-l5bt4\") pod \"odh-model-controller-7998c67dc4-mcpjl\" (UID: \"0492ceb8-72fc-43ea-b95b-45c657ff946d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.621249 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.621178 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv"] Jul 18 13:41:04.623580 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:41:04.623548 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94017353_c705_44dc_b6df_380aa7dbd780.slice/crio-f218237e6a3b93066ed1ff951f6b6879675a9ec3d784b6cc6f37043a910da38e WatchSource:0}: Error finding container f218237e6a3b93066ed1ff951f6b6879675a9ec3d784b6cc6f37043a910da38e: Status 404 returned error can't find the container with id f218237e6a3b93066ed1ff951f6b6879675a9ec3d784b6cc6f37043a910da38e Jul 18 13:41:04.640445 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.640416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:04.781783 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.781755 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mcpjl"] Jul 18 13:41:04.784254 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:41:04.784227 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0492ceb8_72fc_43ea_b95b_45c657ff946d.slice/crio-edcb757b8ac8c3b9dbe175dbd2fdfed94b8761f8b932e37fa342565b11d89dee WatchSource:0}: Error finding container edcb757b8ac8c3b9dbe175dbd2fdfed94b8761f8b932e37fa342565b11d89dee: Status 404 returned error can't find the container with id edcb757b8ac8c3b9dbe175dbd2fdfed94b8761f8b932e37fa342565b11d89dee Jul 18 13:41:04.828017 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.827925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" event={"ID":"94017353-c705-44dc-b6df-380aa7dbd780","Type":"ContainerStarted","Data":"f218237e6a3b93066ed1ff951f6b6879675a9ec3d784b6cc6f37043a910da38e"} Jul 18 13:41:04.829255 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.829231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" event={"ID":"0492ceb8-72fc-43ea-b95b-45c657ff946d","Type":"ContainerStarted","Data":"edcb757b8ac8c3b9dbe175dbd2fdfed94b8761f8b932e37fa342565b11d89dee"} Jul 18 13:41:04.839809 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.839782 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w8txs"] Jul 18 13:41:04.844297 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.844273 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:04.846999 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.846979 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 18 13:41:04.859006 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.858980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w8txs"] Jul 18 13:41:04.927717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.927682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac34956-9721-4e24-9f84-87fde2c723f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:04.927907 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:04.927778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fth\" (UniqueName: \"kubernetes.io/projected/fac34956-9721-4e24-9f84-87fde2c723f7-kube-api-access-t4fth\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.028636 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.028602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fth\" (UniqueName: \"kubernetes.io/projected/fac34956-9721-4e24-9f84-87fde2c723f7-kube-api-access-t4fth\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.028845 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.028728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac34956-9721-4e24-9f84-87fde2c723f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.028935 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:41:05.028869 2564 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 18 13:41:05.028998 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:41:05.028946 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac34956-9721-4e24-9f84-87fde2c723f7-tls-certs podName:fac34956-9721-4e24-9f84-87fde2c723f7 nodeName:}" failed. No retries permitted until 2026-07-18 13:41:05.528922513 +0000 UTC m=+516.078487172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fac34956-9721-4e24-9f84-87fde2c723f7-tls-certs") pod "model-serving-api-54d89bd5fc-w8txs" (UID: "fac34956-9721-4e24-9f84-87fde2c723f7") : secret "model-serving-api-tls" not found Jul 18 13:41:05.045106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.045027 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fth\" (UniqueName: \"kubernetes.io/projected/fac34956-9721-4e24-9f84-87fde2c723f7-kube-api-access-t4fth\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.270364 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.270324 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq"] Jul 18 13:41:05.273894 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.273869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.276862 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.276655 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 18 13:41:05.276862 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.276687 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 18 13:41:05.276862 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.276732 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 13:41:05.276862 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.276694 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 18 13:41:05.276862 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.276667 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gpjhj\"" Jul 18 13:41:05.286294 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.286271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq"] Jul 18 13:41:05.331806 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.331720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.331806 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.331756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56f86\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-kube-api-access-56f86\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.332028 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.331902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-local-certs\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.332028 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.331952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.332028 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.332014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-token\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.332152 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.332042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-cacerts\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.332152 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.332062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433418 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-local-certs\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-token\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-cacerts\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433598 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.433855 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.433613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56f86\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-kube-api-access-56f86\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.434290 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.434259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.436354 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.436211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-local-certs\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.436354 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.436284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.436543 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.436427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-cacerts\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.436701 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.436679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.447738 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.447688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56f86\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-kube-api-access-56f86\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.447961 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.447941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-token\") pod \"istiod-openshift-gateway-7574459656-x5ncq\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.534403 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.534350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac34956-9721-4e24-9f84-87fde2c723f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.537259 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.537212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac34956-9721-4e24-9f84-87fde2c723f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-w8txs\" (UID: \"fac34956-9721-4e24-9f84-87fde2c723f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.586129 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.586049 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:05.737085 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.737057 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq"] Jul 18 13:41:05.739351 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:41:05.739325 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7173c7_a42d_438d_b8af_d2f36ae7c01b.slice/crio-80c93ba07b29cbbe1b67dfca9bef7ff56bed9094d85d57b0061e6137c77c5829 WatchSource:0}: Error finding container 80c93ba07b29cbbe1b67dfca9bef7ff56bed9094d85d57b0061e6137c77c5829: Status 404 returned error can't find the container with id 80c93ba07b29cbbe1b67dfca9bef7ff56bed9094d85d57b0061e6137c77c5829 Jul 18 13:41:05.754367 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.754339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:05.840687 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.840553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" event={"ID":"bf7173c7-a42d-438d-b8af-d2f36ae7c01b","Type":"ContainerStarted","Data":"80c93ba07b29cbbe1b67dfca9bef7ff56bed9094d85d57b0061e6137c77c5829"} Jul 18 13:41:05.925120 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:05.925080 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w8txs"] Jul 18 13:41:05.928858 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:41:05.928824 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac34956_9721_4e24_9f84_87fde2c723f7.slice/crio-dd8c96c0c646b8040344d306f3c43615b8dee8193475092caf92e320ffd4fc06 WatchSource:0}: Error finding container dd8c96c0c646b8040344d306f3c43615b8dee8193475092caf92e320ffd4fc06: Status 404 returned error can't find the container with id dd8c96c0c646b8040344d306f3c43615b8dee8193475092caf92e320ffd4fc06 Jul 18 13:41:06.847959 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:06.847910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" event={"ID":"fac34956-9721-4e24-9f84-87fde2c723f7","Type":"ContainerStarted","Data":"dd8c96c0c646b8040344d306f3c43615b8dee8193475092caf92e320ffd4fc06"} Jul 18 13:41:09.864277 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.864221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" event={"ID":"fac34956-9721-4e24-9f84-87fde2c723f7","Type":"ContainerStarted","Data":"b2310a34abd0dd3528a0f86781a24ba121795cdce2a8d2560aba27d80d739843"} Jul 18 13:41:09.864761 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.864305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:09.866111 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.866083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" event={"ID":"94017353-c705-44dc-b6df-380aa7dbd780","Type":"ContainerStarted","Data":"fec590b276d7f2f20b1be9bdb31ff1b58f4b26340e6d0138c4fca5e065347949"} Jul 18 13:41:09.873485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.873447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" event={"ID":"0492ceb8-72fc-43ea-b95b-45c657ff946d","Type":"ContainerStarted","Data":"bb21f113f6543554066009dc1450ccb968665fbc45ea5d5fec85337156ea9e65"} Jul 18 13:41:09.873650 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.873628 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:09.882996 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.882945 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" podStartSLOduration=2.856864763 podStartE2EDuration="5.882929389s" totalImagesPullingTime="3.026064626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:41:09.880704856 +0000 UTC m=+520.430269537" watchObservedRunningTime="2026-07-18 13:41:09.882929389 +0000 UTC m=+520.432494069" Jul 18 13:41:09.896930 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.896858 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d65948f8d-qvnzv" podStartSLOduration=1.571115189 podStartE2EDuration="5.89683934s" totalImagesPullingTime="4.325724151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:41:09.89541075 +0000 UTC m=+520.444975446" watchObservedRunningTime="2026-07-18 13:41:09.89683934 +0000 UTC m=+520.446404037" Jul 18 13:41:09.911764 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:09.911673 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" podStartSLOduration=1.736268632 podStartE2EDuration="5.911652078s" totalImagesPullingTime="4.175383446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:41:09.909732926 +0000 UTC m=+520.459297602" watchObservedRunningTime="2026-07-18 13:41:09.911652078 +0000 UTC m=+520.461216760" Jul 18 13:41:10.292938 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:10.292897 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:41:10.293056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:10.292965 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:41:10.880695 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:10.880651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" event={"ID":"bf7173c7-a42d-438d-b8af-d2f36ae7c01b","Type":"ContainerStarted","Data":"233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5"} Jul 18 13:41:10.881188 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:10.880903 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:10.932074 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:10.931999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" podStartSLOduration=1.380674771 podStartE2EDuration="5.931980266s" totalImagesPullingTime="4.551305495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:41:10.930020209 +0000 UTC m=+521.479584893" watchObservedRunningTime="2026-07-18 13:41:10.931980266 +0000 UTC m=+521.481544946" Jul 18 13:41:11.887260 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:11.887230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:14.268398 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.268344 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7"] Jul 18 13:41:14.275653 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.275624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.280249 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.280225 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 18 13:41:14.280405 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.280322 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bldpc\"" Jul 18 13:41:14.297226 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.297197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7"] Jul 18 13:41:14.315413 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.315359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ll6\" (UniqueName: \"kubernetes.io/projected/1d32444b-58be-4066-b0d0-bd3451756721-kube-api-access-h4ll6\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.315550 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.315416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d32444b-58be-4066-b0d0-bd3451756721-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.416364 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.416295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ll6\" (UniqueName: \"kubernetes.io/projected/1d32444b-58be-4066-b0d0-bd3451756721-kube-api-access-h4ll6\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.416364 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.416369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d32444b-58be-4066-b0d0-bd3451756721-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.416641 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:41:14.416530 2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 18 13:41:14.416641 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:41:14.416594 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d32444b-58be-4066-b0d0-bd3451756721-cert podName:1d32444b-58be-4066-b0d0-bd3451756721 nodeName:}" failed. No retries permitted until 2026-07-18 13:41:14.916572375 +0000 UTC m=+525.466137034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d32444b-58be-4066-b0d0-bd3451756721-cert") pod "kserve-controller-manager-d6f4b65b8-pfrd7" (UID: "1d32444b-58be-4066-b0d0-bd3451756721") : secret "kserve-webhook-server-cert" not found Jul 18 13:41:14.429263 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.429228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ll6\" (UniqueName: \"kubernetes.io/projected/1d32444b-58be-4066-b0d0-bd3451756721-kube-api-access-h4ll6\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.921049 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.921014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d32444b-58be-4066-b0d0-bd3451756721-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:14.923497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:14.923474 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d32444b-58be-4066-b0d0-bd3451756721-cert\") pod \"kserve-controller-manager-d6f4b65b8-pfrd7\" (UID: \"1d32444b-58be-4066-b0d0-bd3451756721\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:15.185736 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:15.185704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:15.339786 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:15.339761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7"] Jul 18 13:41:15.341800 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:41:15.341771 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d32444b_58be_4066_b0d0_bd3451756721.slice/crio-d3af6a56fbb293a5a10532ddd4aebee131b6690d877f0b6d65efe707e6bf5174 WatchSource:0}: Error finding container d3af6a56fbb293a5a10532ddd4aebee131b6690d877f0b6d65efe707e6bf5174: Status 404 returned error can't find the container with id d3af6a56fbb293a5a10532ddd4aebee131b6690d877f0b6d65efe707e6bf5174 Jul 18 13:41:15.906309 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:15.906277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" event={"ID":"1d32444b-58be-4066-b0d0-bd3451756721","Type":"ContainerStarted","Data":"d3af6a56fbb293a5a10532ddd4aebee131b6690d877f0b6d65efe707e6bf5174"} Jul 18 13:41:17.877314 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.877276 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq"] Jul 18 13:41:17.877714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.877490 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" podUID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" containerName="discovery" containerID="cri-o://233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5" gracePeriod=30 Jul 18 13:41:17.886290 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.886262 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-x5ncq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:8080/ready\": dial tcp 10.133.0.27:8080: connect: connection refused" start-of-body= Jul 18 13:41:17.886449 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.886314 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" podUID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.27:8080/ready\": dial tcp 10.133.0.27:8080: connect: connection refused" Jul 18 13:41:17.915969 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.915933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" event={"ID":"1d32444b-58be-4066-b0d0-bd3451756721","Type":"ContainerStarted","Data":"e6e84bef2e8f9d74706cdbe9e5762eeeaf94b35e83497b2c573b7888f9f4eb44"} Jul 18 13:41:17.916108 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.916013 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:17.933009 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:17.932959 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" podStartSLOduration=1.8554499290000002 podStartE2EDuration="3.932943147s" totalImagesPullingTime="2.077493218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:41:17.931618111 +0000 UTC m=+528.481182792" watchObservedRunningTime="2026-07-18 13:41:17.932943147 +0000 UTC m=+528.482507827" Jul 18 13:41:18.124806 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.124778 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:18.251433 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-dns-cert\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.251433 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-token\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.251689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251466 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-local-certs\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.251689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251487 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-kubeconfig\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.251689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251530 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56f86\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-kube-api-access-56f86\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.251689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-cacerts\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.251689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.251602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-ca-configmap\") pod \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\" (UID: \"bf7173c7-a42d-438d-b8af-d2f36ae7c01b\") " Jul 18 13:41:18.252131 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.252098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:41:18.253892 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.253863 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-cacerts" (OuterVolumeSpecName: "cacerts") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:41:18.253892 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.253879 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:41:18.254156 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.254127 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:41:18.254264 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.254162 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-kube-api-access-56f86" (OuterVolumeSpecName: "kube-api-access-56f86") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "kube-api-access-56f86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:41:18.254264 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.254212 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:41:18.254264 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.254227 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-local-certs" (OuterVolumeSpecName: "local-certs") pod "bf7173c7-a42d-438d-b8af-d2f36ae7c01b" (UID: "bf7173c7-a42d-438d-b8af-d2f36ae7c01b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:41:18.352559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352513 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-ca-configmap\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.352559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352556 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-csr-dns-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.352559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352570 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.352559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352580 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-local-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.352828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352588 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-istio-kubeconfig\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.352828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352596 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56f86\" (UniqueName: \"kubernetes.io/projected/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-kube-api-access-56f86\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.352828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.352604 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf7173c7-a42d-438d-b8af-d2f36ae7c01b-cacerts\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:41:18.921734 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.921691 2564 generic.go:356] "Generic (PLEG): container finished" podID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" containerID="233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5" exitCode=0 Jul 18 13:41:18.922211 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.921760 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" Jul 18 13:41:18.922211 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.921786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" event={"ID":"bf7173c7-a42d-438d-b8af-d2f36ae7c01b","Type":"ContainerDied","Data":"233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5"} Jul 18 13:41:18.922211 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.921890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq" event={"ID":"bf7173c7-a42d-438d-b8af-d2f36ae7c01b","Type":"ContainerDied","Data":"80c93ba07b29cbbe1b67dfca9bef7ff56bed9094d85d57b0061e6137c77c5829"} Jul 18 13:41:18.922211 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.921921 2564 scope.go:117] "RemoveContainer" containerID="233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5" Jul 18 13:41:18.931218 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.931197 2564 scope.go:117] "RemoveContainer" containerID="233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5" Jul 18 13:41:18.931512 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:41:18.931490 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5\": container with ID starting with 233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5 not found: ID does not exist" containerID="233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5" Jul 18 13:41:18.931586 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.931520 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5"} err="failed to get container status \"233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5\": rpc error: code = NotFound desc = could not find container \"233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5\": container with ID starting with 233c9c7b9a006fe8504430b476b719452854aa627b9a2417b7412692201e65b5 not found: ID does not exist" Jul 18 13:41:18.961481 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.961448 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq"] Jul 18 13:41:18.978367 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:18.978333 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-x5ncq"] Jul 18 13:41:19.973152 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:19.973119 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" path="/var/lib/kubelet/pods/bf7173c7-a42d-438d-b8af-d2f36ae7c01b/volumes" Jul 18 13:41:20.882857 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:20.882825 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mcpjl" Jul 18 13:41:20.884843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:20.884820 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-w8txs" Jul 18 13:41:48.927032 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:48.926996 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pfrd7" Jul 18 13:41:59.981706 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.981675 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xgqzn"] Jul 18 13:41:59.982097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.981983 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" containerName="discovery" Jul 18 13:41:59.982097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.981995 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" containerName="discovery" Jul 18 13:41:59.982097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.982043 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7173c7-a42d-438d-b8af-d2f36ae7c01b" containerName="discovery" Jul 18 13:41:59.984853 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.984832 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:41:59.987597 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.987573 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cpl9h\"" Jul 18 13:41:59.988596 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.988573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 13:41:59.988596 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.988592 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 13:41:59.995176 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:41:59.995145 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xgqzn"] Jul 18 13:42:00.026051 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.026007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47pv\" (UniqueName: \"kubernetes.io/projected/d64cc2a2-8126-4e07-8dcb-5770bbc2f042-kube-api-access-m47pv\") pod \"kuadrant-operator-catalog-xgqzn\" (UID: \"d64cc2a2-8126-4e07-8dcb-5770bbc2f042\") " pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:42:00.127037 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.126998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47pv\" (UniqueName: \"kubernetes.io/projected/d64cc2a2-8126-4e07-8dcb-5770bbc2f042-kube-api-access-m47pv\") pod \"kuadrant-operator-catalog-xgqzn\" (UID: \"d64cc2a2-8126-4e07-8dcb-5770bbc2f042\") " pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:42:00.134814 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.134783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47pv\" (UniqueName: \"kubernetes.io/projected/d64cc2a2-8126-4e07-8dcb-5770bbc2f042-kube-api-access-m47pv\") pod \"kuadrant-operator-catalog-xgqzn\" (UID: \"d64cc2a2-8126-4e07-8dcb-5770bbc2f042\") " pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:42:00.296696 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.296603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:42:00.352389 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.352340 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xgqzn"] Jul 18 13:42:00.430629 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.430586 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xgqzn"] Jul 18 13:42:00.433697 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:00.433665 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64cc2a2_8126_4e07_8dcb_5770bbc2f042.slice/crio-8aef04bd22a4a3f86f3e24eb7bc479572cf7a81a79b6f8bf238529808aa9f1db WatchSource:0}: Error finding container 8aef04bd22a4a3f86f3e24eb7bc479572cf7a81a79b6f8bf238529808aa9f1db: Status 404 returned error can't find the container with id 8aef04bd22a4a3f86f3e24eb7bc479572cf7a81a79b6f8bf238529808aa9f1db Jul 18 13:42:00.558299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.558212 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cmsnz"] Jul 18 13:42:00.562663 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.562645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:00.568750 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.568725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cmsnz"] Jul 18 13:42:00.630727 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.630689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gq7g\" (UniqueName: \"kubernetes.io/projected/30cae6b3-f118-4cee-af3b-e82bf1dbb6be-kube-api-access-9gq7g\") pod \"kuadrant-operator-catalog-cmsnz\" (UID: \"30cae6b3-f118-4cee-af3b-e82bf1dbb6be\") " pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:00.731305 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.731265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gq7g\" (UniqueName: \"kubernetes.io/projected/30cae6b3-f118-4cee-af3b-e82bf1dbb6be-kube-api-access-9gq7g\") pod \"kuadrant-operator-catalog-cmsnz\" (UID: \"30cae6b3-f118-4cee-af3b-e82bf1dbb6be\") " pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:00.739241 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.739215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gq7g\" (UniqueName: \"kubernetes.io/projected/30cae6b3-f118-4cee-af3b-e82bf1dbb6be-kube-api-access-9gq7g\") pod \"kuadrant-operator-catalog-cmsnz\" (UID: \"30cae6b3-f118-4cee-af3b-e82bf1dbb6be\") " pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:00.873099 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:00.873008 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:01.013430 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:01.013407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cmsnz"] Jul 18 13:42:01.017701 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:01.016365 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cae6b3_f118_4cee_af3b_e82bf1dbb6be.slice/crio-9f836eb733370c6886c9b5644a89b775e67782ca425c9a25046fd7158097b96e WatchSource:0}: Error finding container 9f836eb733370c6886c9b5644a89b775e67782ca425c9a25046fd7158097b96e: Status 404 returned error can't find the container with id 9f836eb733370c6886c9b5644a89b775e67782ca425c9a25046fd7158097b96e Jul 18 13:42:01.110716 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:01.110681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" event={"ID":"d64cc2a2-8126-4e07-8dcb-5770bbc2f042","Type":"ContainerStarted","Data":"8aef04bd22a4a3f86f3e24eb7bc479572cf7a81a79b6f8bf238529808aa9f1db"} Jul 18 13:42:01.111819 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:01.111792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" event={"ID":"30cae6b3-f118-4cee-af3b-e82bf1dbb6be","Type":"ContainerStarted","Data":"9f836eb733370c6886c9b5644a89b775e67782ca425c9a25046fd7158097b96e"} Jul 18 13:42:03.123982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.123941 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" event={"ID":"d64cc2a2-8126-4e07-8dcb-5770bbc2f042","Type":"ContainerStarted","Data":"7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142"} Jul 18 13:42:03.124479 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.123964 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" podUID="d64cc2a2-8126-4e07-8dcb-5770bbc2f042" containerName="registry-server" containerID="cri-o://7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142" gracePeriod=2 Jul 18 13:42:03.125534 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.125504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" event={"ID":"30cae6b3-f118-4cee-af3b-e82bf1dbb6be","Type":"ContainerStarted","Data":"b76cf0670973710510b9b366ccfb4e5bd20e4e449a38877e7fbe20aa4501ee22"} Jul 18 13:42:03.140176 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.140127 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" podStartSLOduration=1.890882214 podStartE2EDuration="4.140111612s" totalImagesPullingTime="2.249229398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:03.138096282 +0000 UTC m=+573.687660962" watchObservedRunningTime="2026-07-18 13:42:03.140111612 +0000 UTC m=+573.689676293" Jul 18 13:42:03.153070 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.153011 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" podStartSLOduration=1.453241783 podStartE2EDuration="3.152993024s" totalImagesPullingTime="1.699751241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:03.152792939 +0000 UTC m=+573.702357620" watchObservedRunningTime="2026-07-18 13:42:03.152993024 +0000 UTC m=+573.702557708" Jul 18 13:42:03.360251 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.360223 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:42:03.456431 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.456399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m47pv\" (UniqueName: \"kubernetes.io/projected/d64cc2a2-8126-4e07-8dcb-5770bbc2f042-kube-api-access-m47pv\") pod \"d64cc2a2-8126-4e07-8dcb-5770bbc2f042\" (UID: \"d64cc2a2-8126-4e07-8dcb-5770bbc2f042\") " Jul 18 13:42:03.458559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.458527 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64cc2a2-8126-4e07-8dcb-5770bbc2f042-kube-api-access-m47pv" (OuterVolumeSpecName: "kube-api-access-m47pv") pod "d64cc2a2-8126-4e07-8dcb-5770bbc2f042" (UID: "d64cc2a2-8126-4e07-8dcb-5770bbc2f042"). InnerVolumeSpecName "kube-api-access-m47pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:42:03.557828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:03.557776 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47pv\" (UniqueName: \"kubernetes.io/projected/d64cc2a2-8126-4e07-8dcb-5770bbc2f042-kube-api-access-m47pv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:04.131400 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.131344 2564 generic.go:356] "Generic (PLEG): container finished" podID="d64cc2a2-8126-4e07-8dcb-5770bbc2f042" containerID="7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142" exitCode=0 Jul 18 13:42:04.131877 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.131419 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" Jul 18 13:42:04.131877 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.131431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" event={"ID":"d64cc2a2-8126-4e07-8dcb-5770bbc2f042","Type":"ContainerDied","Data":"7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142"} Jul 18 13:42:04.131877 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.131479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xgqzn" event={"ID":"d64cc2a2-8126-4e07-8dcb-5770bbc2f042","Type":"ContainerDied","Data":"8aef04bd22a4a3f86f3e24eb7bc479572cf7a81a79b6f8bf238529808aa9f1db"} Jul 18 13:42:04.131877 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.131499 2564 scope.go:117] "RemoveContainer" containerID="7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142" Jul 18 13:42:04.140450 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.140433 2564 scope.go:117] "RemoveContainer" containerID="7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142" Jul 18 13:42:04.140731 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:42:04.140711 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142\": container with ID starting with 7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142 not found: ID does not exist" containerID="7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142" Jul 18 13:42:04.140796 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.140745 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142"} err="failed to get container status \"7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142\": rpc error: code = NotFound desc = could not find container \"7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142\": container with ID starting with 7727f9594cefc34c67940ee68bf1c7ee0d07e55f488d75315af51f9de2ee1142 not found: ID does not exist" Jul 18 13:42:04.147679 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.147655 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xgqzn"] Jul 18 13:42:04.151411 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:04.151362 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xgqzn"] Jul 18 13:42:05.972459 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:05.972427 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64cc2a2-8126-4e07-8dcb-5770bbc2f042" path="/var/lib/kubelet/pods/d64cc2a2-8126-4e07-8dcb-5770bbc2f042/volumes" Jul 18 13:42:10.873814 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:10.873777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:10.874244 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:10.873848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:10.897109 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:10.897082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:11.185685 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:11.185662 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-cmsnz" Jul 18 13:42:15.794034 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.792184 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz"] Jul 18 13:42:15.794034 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.792961 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64cc2a2-8126-4e07-8dcb-5770bbc2f042" containerName="registry-server" Jul 18 13:42:15.794034 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.792983 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64cc2a2-8126-4e07-8dcb-5770bbc2f042" containerName="registry-server" Jul 18 13:42:15.794034 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.793137 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64cc2a2-8126-4e07-8dcb-5770bbc2f042" containerName="registry-server" Jul 18 13:42:15.798533 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.798503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.801245 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.801223 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rzx6c\"" Jul 18 13:42:15.801493 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.801444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz"] Jul 18 13:42:15.870985 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.870948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.870985 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.870989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gtm\" (UniqueName: \"kubernetes.io/projected/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-kube-api-access-b4gtm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.871186 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.871016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.972441 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.972409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.972627 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.972460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gtm\" (UniqueName: \"kubernetes.io/projected/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-kube-api-access-b4gtm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.972627 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.972505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.972838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.972819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.972921 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.972849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:15.980790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:15.980761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gtm\" (UniqueName: \"kubernetes.io/projected/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-kube-api-access-b4gtm\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:16.108706 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.108616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:16.239518 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.239489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz"] Jul 18 13:42:16.241669 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:16.241640 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5470b4e_48bc_4c6e_a8a2_ada8d7636789.slice/crio-272a63dca7a0801ccaed2fc57baa3ba259dc790ea4123351e1d88a5c744eeace WatchSource:0}: Error finding container 272a63dca7a0801ccaed2fc57baa3ba259dc790ea4123351e1d88a5c744eeace: Status 404 returned error can't find the container with id 272a63dca7a0801ccaed2fc57baa3ba259dc790ea4123351e1d88a5c744eeace Jul 18 13:42:16.586723 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.586686 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9"] Jul 18 13:42:16.590126 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.590104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.598291 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.598263 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9"] Jul 18 13:42:16.680541 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.680505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.680709 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.680599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7p4\" (UniqueName: \"kubernetes.io/projected/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-kube-api-access-nw7p4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.680709 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.680634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.781906 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.781871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7p4\" (UniqueName: \"kubernetes.io/projected/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-kube-api-access-nw7p4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.782083 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.781918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.782083 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.781948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.782314 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.782296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.782347 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.782325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.790046 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.790021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7p4\" (UniqueName: \"kubernetes.io/projected/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-kube-api-access-nw7p4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:16.900208 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:16.900102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:17.035619 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.035592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9"] Jul 18 13:42:17.037495 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:17.037460 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb345f19e_eadb_4cfa_b1f4_e76c18f3257a.slice/crio-9f2b512e6d5762fc0c785b1a991cb0f92240c20ca7e91df1b216d8a2993147ab WatchSource:0}: Error finding container 9f2b512e6d5762fc0c785b1a991cb0f92240c20ca7e91df1b216d8a2993147ab: Status 404 returned error can't find the container with id 9f2b512e6d5762fc0c785b1a991cb0f92240c20ca7e91df1b216d8a2993147ab Jul 18 13:42:17.191707 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.191677 2564 generic.go:356] "Generic (PLEG): container finished" podID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerID="6ca5b6a4cfaae16d69f171b97806f98a5d50a06b5de5e43ce8752fb9f310661e" exitCode=0 Jul 18 13:42:17.191901 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.191777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" event={"ID":"b345f19e-eadb-4cfa-b1f4-e76c18f3257a","Type":"ContainerDied","Data":"6ca5b6a4cfaae16d69f171b97806f98a5d50a06b5de5e43ce8752fb9f310661e"} Jul 18 13:42:17.191901 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.191837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" event={"ID":"b345f19e-eadb-4cfa-b1f4-e76c18f3257a","Type":"ContainerStarted","Data":"9f2b512e6d5762fc0c785b1a991cb0f92240c20ca7e91df1b216d8a2993147ab"} Jul 18 13:42:17.193344 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.193325 2564 generic.go:356] "Generic (PLEG): container finished" podID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerID="5a40621e3bbf01e0907297ae39fb360e69d2e7207c73bfc464a664f62ecc8999" exitCode=0 Jul 18 13:42:17.193434 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.193390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" event={"ID":"b5470b4e-48bc-4c6e-a8a2-ada8d7636789","Type":"ContainerDied","Data":"5a40621e3bbf01e0907297ae39fb360e69d2e7207c73bfc464a664f62ecc8999"} Jul 18 13:42:17.193434 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.193417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" event={"ID":"b5470b4e-48bc-4c6e-a8a2-ada8d7636789","Type":"ContainerStarted","Data":"272a63dca7a0801ccaed2fc57baa3ba259dc790ea4123351e1d88a5c744eeace"} Jul 18 13:42:17.386850 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.386813 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29"] Jul 18 13:42:17.390294 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.390277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.398929 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.398899 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29"] Jul 18 13:42:17.488354 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.488271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.488354 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.488306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.488354 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.488340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqptd\" (UniqueName: \"kubernetes.io/projected/72b7aeca-8a94-4994-8f83-1f2061afe4c1-kube-api-access-bqptd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.588813 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.588781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.588813 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.588815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.589074 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.588843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqptd\" (UniqueName: \"kubernetes.io/projected/72b7aeca-8a94-4994-8f83-1f2061afe4c1-kube-api-access-bqptd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.589196 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.589173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.589258 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.589210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.596971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.596936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqptd\" (UniqueName: \"kubernetes.io/projected/72b7aeca-8a94-4994-8f83-1f2061afe4c1-kube-api-access-bqptd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.699718 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.699690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:17.841224 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.841100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29"] Jul 18 13:42:17.853150 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:17.853114 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b7aeca_8a94_4994_8f83_1f2061afe4c1.slice/crio-f6b7a8eececfe1941a6131c26c823f316ab8040b1d84c60848bfe69abae1345d WatchSource:0}: Error finding container f6b7a8eececfe1941a6131c26c823f316ab8040b1d84c60848bfe69abae1345d: Status 404 returned error can't find the container with id f6b7a8eececfe1941a6131c26c823f316ab8040b1d84c60848bfe69abae1345d Jul 18 13:42:17.995005 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.994969 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c"] Jul 18 13:42:17.999957 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:17.999933 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.007097 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.007035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c"] Jul 18 13:42:18.092411 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.092291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.092660 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.092627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.092752 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.092730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54xf\" (UniqueName: \"kubernetes.io/projected/79eff395-f999-40bc-96cb-164545194b03-kube-api-access-c54xf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.194014 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.193978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54xf\" (UniqueName: \"kubernetes.io/projected/79eff395-f999-40bc-96cb-164545194b03-kube-api-access-c54xf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.194200 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.194050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.194200 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.194162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.194508 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.194489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.194560 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.194546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.199255 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.199227 2564 generic.go:356] "Generic (PLEG): container finished" podID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerID="e75de3f8e3b940cc280beb10dea54b9d9db6da3f07cac0ea34a07dc67212ac37" exitCode=0 Jul 18 13:42:18.199417 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.199291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" event={"ID":"72b7aeca-8a94-4994-8f83-1f2061afe4c1","Type":"ContainerDied","Data":"e75de3f8e3b940cc280beb10dea54b9d9db6da3f07cac0ea34a07dc67212ac37"} Jul 18 13:42:18.199417 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.199344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" event={"ID":"72b7aeca-8a94-4994-8f83-1f2061afe4c1","Type":"ContainerStarted","Data":"f6b7a8eececfe1941a6131c26c823f316ab8040b1d84c60848bfe69abae1345d"} Jul 18 13:42:18.201444 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.201422 2564 generic.go:356] "Generic (PLEG): container finished" podID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerID="08422e33501556861a68ef51261aafbc8b7218187166d5d17633de19cc98ba7d" exitCode=0 Jul 18 13:42:18.201527 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.201471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" event={"ID":"b5470b4e-48bc-4c6e-a8a2-ada8d7636789","Type":"ContainerDied","Data":"08422e33501556861a68ef51261aafbc8b7218187166d5d17633de19cc98ba7d"} Jul 18 13:42:18.203485 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.203464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54xf\" (UniqueName: \"kubernetes.io/projected/79eff395-f999-40bc-96cb-164545194b03-kube-api-access-c54xf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.310852 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.310820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:18.442933 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:18.442905 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c"] Jul 18 13:42:18.444483 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:18.444456 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79eff395_f999_40bc_96cb_164545194b03.slice/crio-5e27400343d79c2983d06925deea343f76b03b3d871c6214e9393b2a77a85e36 WatchSource:0}: Error finding container 5e27400343d79c2983d06925deea343f76b03b3d871c6214e9393b2a77a85e36: Status 404 returned error can't find the container with id 5e27400343d79c2983d06925deea343f76b03b3d871c6214e9393b2a77a85e36 Jul 18 13:42:19.209169 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.209133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" event={"ID":"72b7aeca-8a94-4994-8f83-1f2061afe4c1","Type":"ContainerStarted","Data":"9afcce1a57ba14eb3ddfea63f4f832dc7f600e18aaf7bd8c354d40ff03ca5bee"} Jul 18 13:42:19.210737 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.210707 2564 generic.go:356] "Generic (PLEG): container finished" podID="79eff395-f999-40bc-96cb-164545194b03" containerID="7cf29a75dd46f83402a279c7a9dbb623c8dfa892a4b87c953af925694d763819" exitCode=0 Jul 18 13:42:19.210872 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.210795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" event={"ID":"79eff395-f999-40bc-96cb-164545194b03","Type":"ContainerDied","Data":"7cf29a75dd46f83402a279c7a9dbb623c8dfa892a4b87c953af925694d763819"} Jul 18 13:42:19.210872 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.210850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" event={"ID":"79eff395-f999-40bc-96cb-164545194b03","Type":"ContainerStarted","Data":"5e27400343d79c2983d06925deea343f76b03b3d871c6214e9393b2a77a85e36"} Jul 18 13:42:19.212949 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.212921 2564 generic.go:356] "Generic (PLEG): container finished" podID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerID="b60095cac48a1005ef1ec7d341404f02b78e5029b6aebf23daf1c30e08892351" exitCode=0 Jul 18 13:42:19.213117 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.213097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" event={"ID":"b345f19e-eadb-4cfa-b1f4-e76c18f3257a","Type":"ContainerDied","Data":"b60095cac48a1005ef1ec7d341404f02b78e5029b6aebf23daf1c30e08892351"} Jul 18 13:42:19.215611 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.215590 2564 generic.go:356] "Generic (PLEG): container finished" podID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerID="6e9a3a53d09d85b37ff5a896f9024dd7866623aefb3facb90f2e5ca28cd2a59c" exitCode=0 Jul 18 13:42:19.215690 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:19.215643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" event={"ID":"b5470b4e-48bc-4c6e-a8a2-ada8d7636789","Type":"ContainerDied","Data":"6e9a3a53d09d85b37ff5a896f9024dd7866623aefb3facb90f2e5ca28cd2a59c"} Jul 18 13:42:20.222244 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.222208 2564 generic.go:356] "Generic (PLEG): container finished" podID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerID="9afcce1a57ba14eb3ddfea63f4f832dc7f600e18aaf7bd8c354d40ff03ca5bee" exitCode=0 Jul 18 13:42:20.222739 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.222301 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" event={"ID":"72b7aeca-8a94-4994-8f83-1f2061afe4c1","Type":"ContainerDied","Data":"9afcce1a57ba14eb3ddfea63f4f832dc7f600e18aaf7bd8c354d40ff03ca5bee"} Jul 18 13:42:20.224189 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.224167 2564 generic.go:356] "Generic (PLEG): container finished" podID="79eff395-f999-40bc-96cb-164545194b03" containerID="109438b92cd42ec2a0839fca06a50902f5e406fc06c89e4110b57e63507d92b5" exitCode=0 Jul 18 13:42:20.224282 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.224259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" event={"ID":"79eff395-f999-40bc-96cb-164545194b03","Type":"ContainerDied","Data":"109438b92cd42ec2a0839fca06a50902f5e406fc06c89e4110b57e63507d92b5"} Jul 18 13:42:20.226509 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.226487 2564 generic.go:356] "Generic (PLEG): container finished" podID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerID="482eb01157da44c707b44337f0ee66d3064028cdfce69f081860d6de4eab022c" exitCode=0 Jul 18 13:42:20.226591 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.226536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" event={"ID":"b345f19e-eadb-4cfa-b1f4-e76c18f3257a","Type":"ContainerDied","Data":"482eb01157da44c707b44337f0ee66d3064028cdfce69f081860d6de4eab022c"} Jul 18 13:42:20.376741 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.376713 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:20.515603 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.515517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4gtm\" (UniqueName: \"kubernetes.io/projected/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-kube-api-access-b4gtm\") pod \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " Jul 18 13:42:20.515603 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.515569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-bundle\") pod \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " Jul 18 13:42:20.515769 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.515689 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-util\") pod \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\" (UID: \"b5470b4e-48bc-4c6e-a8a2-ada8d7636789\") " Jul 18 13:42:20.516188 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.516155 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-bundle" (OuterVolumeSpecName: "bundle") pod "b5470b4e-48bc-4c6e-a8a2-ada8d7636789" (UID: "b5470b4e-48bc-4c6e-a8a2-ada8d7636789"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:20.517662 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.517640 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-kube-api-access-b4gtm" (OuterVolumeSpecName: "kube-api-access-b4gtm") pod "b5470b4e-48bc-4c6e-a8a2-ada8d7636789" (UID: "b5470b4e-48bc-4c6e-a8a2-ada8d7636789"). InnerVolumeSpecName "kube-api-access-b4gtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:42:20.521509 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.521477 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-util" (OuterVolumeSpecName: "util") pod "b5470b4e-48bc-4c6e-a8a2-ada8d7636789" (UID: "b5470b4e-48bc-4c6e-a8a2-ada8d7636789"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:20.617334 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.617285 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gtm\" (UniqueName: \"kubernetes.io/projected/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-kube-api-access-b4gtm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:20.617334 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.617330 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:20.617334 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:20.617339 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5470b4e-48bc-4c6e-a8a2-ada8d7636789-util\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:21.233544 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.233513 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" Jul 18 13:42:21.233544 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.233526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz" event={"ID":"b5470b4e-48bc-4c6e-a8a2-ada8d7636789","Type":"ContainerDied","Data":"272a63dca7a0801ccaed2fc57baa3ba259dc790ea4123351e1d88a5c744eeace"} Jul 18 13:42:21.233993 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.233613 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="272a63dca7a0801ccaed2fc57baa3ba259dc790ea4123351e1d88a5c744eeace" Jul 18 13:42:21.235620 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.235550 2564 generic.go:356] "Generic (PLEG): container finished" podID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerID="f135838a3e756d7f64d0b7c405375935c1616cf4d00663be2c488218ada23e03" exitCode=0 Jul 18 13:42:21.235745 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.235619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" event={"ID":"72b7aeca-8a94-4994-8f83-1f2061afe4c1","Type":"ContainerDied","Data":"f135838a3e756d7f64d0b7c405375935c1616cf4d00663be2c488218ada23e03"} Jul 18 13:42:21.237614 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.237588 2564 generic.go:356] "Generic (PLEG): container finished" podID="79eff395-f999-40bc-96cb-164545194b03" containerID="0441108664f488a244003f3df6ea18d2180901394789a9cd7d67bfd37ce8cbc0" exitCode=0 Jul 18 13:42:21.237708 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.237650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" event={"ID":"79eff395-f999-40bc-96cb-164545194b03","Type":"ContainerDied","Data":"0441108664f488a244003f3df6ea18d2180901394789a9cd7d67bfd37ce8cbc0"} Jul 18 13:42:21.375232 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.375201 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:21.526143 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.526050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw7p4\" (UniqueName: \"kubernetes.io/projected/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-kube-api-access-nw7p4\") pod \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " Jul 18 13:42:21.526143 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.526131 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-bundle\") pod \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " Jul 18 13:42:21.526331 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.526305 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-util\") pod \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\" (UID: \"b345f19e-eadb-4cfa-b1f4-e76c18f3257a\") " Jul 18 13:42:21.526630 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.526600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-bundle" (OuterVolumeSpecName: "bundle") pod "b345f19e-eadb-4cfa-b1f4-e76c18f3257a" (UID: "b345f19e-eadb-4cfa-b1f4-e76c18f3257a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:21.528244 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.528222 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-kube-api-access-nw7p4" (OuterVolumeSpecName: "kube-api-access-nw7p4") pod "b345f19e-eadb-4cfa-b1f4-e76c18f3257a" (UID: "b345f19e-eadb-4cfa-b1f4-e76c18f3257a"). InnerVolumeSpecName "kube-api-access-nw7p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:42:21.531153 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.531129 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-util" (OuterVolumeSpecName: "util") pod "b345f19e-eadb-4cfa-b1f4-e76c18f3257a" (UID: "b345f19e-eadb-4cfa-b1f4-e76c18f3257a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:21.627812 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.627775 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7p4\" (UniqueName: \"kubernetes.io/projected/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-kube-api-access-nw7p4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:21.627812 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.627807 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:21.627812 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:21.627816 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b345f19e-eadb-4cfa-b1f4-e76c18f3257a-util\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:22.245624 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.245538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" event={"ID":"b345f19e-eadb-4cfa-b1f4-e76c18f3257a","Type":"ContainerDied","Data":"9f2b512e6d5762fc0c785b1a991cb0f92240c20ca7e91df1b216d8a2993147ab"} Jul 18 13:42:22.245624 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.245573 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9" Jul 18 13:42:22.245624 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.245599 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f2b512e6d5762fc0c785b1a991cb0f92240c20ca7e91df1b216d8a2993147ab" Jul 18 13:42:22.375853 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.375828 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:22.403991 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.403967 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:22.538507 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.538410 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-util\") pod \"79eff395-f999-40bc-96cb-164545194b03\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " Jul 18 13:42:22.538507 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.538455 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqptd\" (UniqueName: \"kubernetes.io/projected/72b7aeca-8a94-4994-8f83-1f2061afe4c1-kube-api-access-bqptd\") pod \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " Jul 18 13:42:22.538507 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.538477 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-util\") pod \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " Jul 18 13:42:22.538781 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.538517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-bundle\") pod \"79eff395-f999-40bc-96cb-164545194b03\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " Jul 18 13:42:22.538781 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.538563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c54xf\" (UniqueName: \"kubernetes.io/projected/79eff395-f999-40bc-96cb-164545194b03-kube-api-access-c54xf\") pod \"79eff395-f999-40bc-96cb-164545194b03\" (UID: \"79eff395-f999-40bc-96cb-164545194b03\") " Jul 18 13:42:22.538781 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.538597 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-bundle\") pod \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\" (UID: \"72b7aeca-8a94-4994-8f83-1f2061afe4c1\") " Jul 18 13:42:22.539111 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.539079 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-bundle" (OuterVolumeSpecName: "bundle") pod "79eff395-f999-40bc-96cb-164545194b03" (UID: "79eff395-f999-40bc-96cb-164545194b03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:22.539470 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.539438 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-bundle" (OuterVolumeSpecName: "bundle") pod "72b7aeca-8a94-4994-8f83-1f2061afe4c1" (UID: "72b7aeca-8a94-4994-8f83-1f2061afe4c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:22.540724 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.540688 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b7aeca-8a94-4994-8f83-1f2061afe4c1-kube-api-access-bqptd" (OuterVolumeSpecName: "kube-api-access-bqptd") pod "72b7aeca-8a94-4994-8f83-1f2061afe4c1" (UID: "72b7aeca-8a94-4994-8f83-1f2061afe4c1"). InnerVolumeSpecName "kube-api-access-bqptd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:42:22.540826 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.540800 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eff395-f999-40bc-96cb-164545194b03-kube-api-access-c54xf" (OuterVolumeSpecName: "kube-api-access-c54xf") pod "79eff395-f999-40bc-96cb-164545194b03" (UID: "79eff395-f999-40bc-96cb-164545194b03"). InnerVolumeSpecName "kube-api-access-c54xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:42:22.544299 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.544276 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-util" (OuterVolumeSpecName: "util") pod "72b7aeca-8a94-4994-8f83-1f2061afe4c1" (UID: "72b7aeca-8a94-4994-8f83-1f2061afe4c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:22.546114 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.546090 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-util" (OuterVolumeSpecName: "util") pod "79eff395-f999-40bc-96cb-164545194b03" (UID: "79eff395-f999-40bc-96cb-164545194b03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:42:22.639971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.639933 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-util\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:22.639971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.639965 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqptd\" (UniqueName: \"kubernetes.io/projected/72b7aeca-8a94-4994-8f83-1f2061afe4c1-kube-api-access-bqptd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:22.639971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.639974 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-util\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:22.639971 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.639984 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79eff395-f999-40bc-96cb-164545194b03-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:22.640220 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.639993 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54xf\" (UniqueName: \"kubernetes.io/projected/79eff395-f999-40bc-96cb-164545194b03-kube-api-access-c54xf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:22.640220 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:22.640000 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72b7aeca-8a94-4994-8f83-1f2061afe4c1-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:42:23.252061 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:23.252036 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" Jul 18 13:42:23.252464 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:23.252029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29" event={"ID":"72b7aeca-8a94-4994-8f83-1f2061afe4c1","Type":"ContainerDied","Data":"f6b7a8eececfe1941a6131c26c823f316ab8040b1d84c60848bfe69abae1345d"} Jul 18 13:42:23.252464 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:23.252160 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b7a8eececfe1941a6131c26c823f316ab8040b1d84c60848bfe69abae1345d" Jul 18 13:42:23.254062 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:23.254044 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" Jul 18 13:42:23.254155 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:23.254040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c" event={"ID":"79eff395-f999-40bc-96cb-164545194b03","Type":"ContainerDied","Data":"5e27400343d79c2983d06925deea343f76b03b3d871c6214e9393b2a77a85e36"} Jul 18 13:42:23.254155 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:23.254151 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e27400343d79c2983d06925deea343f76b03b3d871c6214e9393b2a77a85e36" Jul 18 13:42:31.654415 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654363 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd8c85fc8-2kwrs"] Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654691 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654703 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654712 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654718 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654730 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654736 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654742 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654747 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654754 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654758 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654765 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654770 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654778 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654784 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654790 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654795 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654801 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654806 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="util" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654811 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654816 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654829 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654834 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654840 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654846 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="pull" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654906 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79eff395-f999-40bc-96cb-164545194b03" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654914 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5470b4e-48bc-4c6e-a8a2-ada8d7636789" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654919 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b7aeca-8a94-4994-8f83-1f2061afe4c1" containerName="extract" Jul 18 13:42:31.656838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.654927 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b345f19e-eadb-4cfa-b1f4-e76c18f3257a" containerName="extract" Jul 18 13:42:31.657705 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.657689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.667566 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.667536 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd8c85fc8-2kwrs"] Jul 18 13:42:31.718263 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-trusted-ca-bundle\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.718497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-service-ca\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.718497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncvq\" (UniqueName: \"kubernetes.io/projected/64d2798c-7479-4a95-983e-829c2f79831b-kube-api-access-pncvq\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.718497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d2798c-7479-4a95-983e-829c2f79831b-console-serving-cert\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.718497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-console-config\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.718497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d2798c-7479-4a95-983e-829c2f79831b-console-oauth-config\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.718699 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.718506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-oauth-serving-cert\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819319 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d2798c-7479-4a95-983e-829c2f79831b-console-serving-cert\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819319 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-console-config\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819599 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d2798c-7479-4a95-983e-829c2f79831b-console-oauth-config\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819599 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-oauth-serving-cert\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819599 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-trusted-ca-bundle\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819599 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-service-ca\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.819599 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.819559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncvq\" (UniqueName: \"kubernetes.io/projected/64d2798c-7479-4a95-983e-829c2f79831b-kube-api-access-pncvq\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.820187 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.820156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-console-config\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.820309 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.820156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-oauth-serving-cert\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.820309 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.820240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-service-ca\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.820461 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.820325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d2798c-7479-4a95-983e-829c2f79831b-trusted-ca-bundle\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.822018 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.821991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d2798c-7479-4a95-983e-829c2f79831b-console-oauth-config\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.822118 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.822050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d2798c-7479-4a95-983e-829c2f79831b-console-serving-cert\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.827643 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.827622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncvq\" (UniqueName: \"kubernetes.io/projected/64d2798c-7479-4a95-983e-829c2f79831b-kube-api-access-pncvq\") pod \"console-fd8c85fc8-2kwrs\" (UID: \"64d2798c-7479-4a95-983e-829c2f79831b\") " pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:31.967179 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:31.967143 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:32.105871 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.105833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd8c85fc8-2kwrs"] Jul 18 13:42:32.108538 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:32.108500 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d2798c_7479_4a95_983e_829c2f79831b.slice/crio-00f85d6f3ffbbb9657700377ae0ce96384e5c692ec13f913872403cf6980e2e5 WatchSource:0}: Error finding container 00f85d6f3ffbbb9657700377ae0ce96384e5c692ec13f913872403cf6980e2e5: Status 404 returned error can't find the container with id 00f85d6f3ffbbb9657700377ae0ce96384e5c692ec13f913872403cf6980e2e5 Jul 18 13:42:32.299947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.299858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8c85fc8-2kwrs" event={"ID":"64d2798c-7479-4a95-983e-829c2f79831b","Type":"ContainerStarted","Data":"3f889e0adf27287b4c7450364d9ffb650b263b7e9ece105ae7f87794eed06e4a"} Jul 18 13:42:32.299947 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.299910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8c85fc8-2kwrs" event={"ID":"64d2798c-7479-4a95-983e-829c2f79831b","Type":"ContainerStarted","Data":"00f85d6f3ffbbb9657700377ae0ce96384e5c692ec13f913872403cf6980e2e5"} Jul 18 13:42:32.318423 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.318278 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd8c85fc8-2kwrs" podStartSLOduration=1.318256068 podStartE2EDuration="1.318256068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:32.318120301 +0000 UTC m=+602.867684983" watchObservedRunningTime="2026-07-18 13:42:32.318256068 +0000 UTC m=+602.867820750" Jul 18 13:42:32.953273 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.953219 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn"] Jul 18 13:42:32.956645 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.956625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:32.959341 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.959319 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8bnlf\"" Jul 18 13:42:32.959473 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.959364 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 18 13:42:32.971305 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:32.971279 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn"] Jul 18 13:42:33.027846 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:33.027810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxcf\" (UniqueName: \"kubernetes.io/projected/3998ee96-6963-47d3-9d14-c52a002aa7ad-kube-api-access-nwxcf\") pod \"dns-operator-controller-manager-864dd78c4c-tmnxn\" (UID: \"3998ee96-6963-47d3-9d14-c52a002aa7ad\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:33.129155 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:33.129121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxcf\" (UniqueName: \"kubernetes.io/projected/3998ee96-6963-47d3-9d14-c52a002aa7ad-kube-api-access-nwxcf\") pod \"dns-operator-controller-manager-864dd78c4c-tmnxn\" (UID: \"3998ee96-6963-47d3-9d14-c52a002aa7ad\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:33.140546 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:33.140518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxcf\" (UniqueName: \"kubernetes.io/projected/3998ee96-6963-47d3-9d14-c52a002aa7ad-kube-api-access-nwxcf\") pod \"dns-operator-controller-manager-864dd78c4c-tmnxn\" (UID: \"3998ee96-6963-47d3-9d14-c52a002aa7ad\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:33.267225 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:33.267128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:33.428515 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:33.428485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn"] Jul 18 13:42:33.430765 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:33.430739 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3998ee96_6963_47d3_9d14_c52a002aa7ad.slice/crio-cd158f88ef097d349111497a105ff98f570d362b5a50073d466d6bac2d2679b9 WatchSource:0}: Error finding container cd158f88ef097d349111497a105ff98f570d362b5a50073d466d6bac2d2679b9: Status 404 returned error can't find the container with id cd158f88ef097d349111497a105ff98f570d362b5a50073d466d6bac2d2679b9 Jul 18 13:42:34.310773 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:34.310741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" event={"ID":"3998ee96-6963-47d3-9d14-c52a002aa7ad","Type":"ContainerStarted","Data":"cd158f88ef097d349111497a105ff98f570d362b5a50073d466d6bac2d2679b9"} Jul 18 13:42:36.323144 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:36.323105 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" event={"ID":"3998ee96-6963-47d3-9d14-c52a002aa7ad","Type":"ContainerStarted","Data":"52a2f2d84d6fe685439d3a23fb07accc76ca3bcf8ce2e312779b7689a60ba00a"} Jul 18 13:42:36.323565 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:36.323184 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:36.341502 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:36.341449 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" podStartSLOduration=2.388570652 podStartE2EDuration="4.341431572s" totalImagesPullingTime="1.95286092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:36.340650559 +0000 UTC m=+606.890215240" watchObservedRunningTime="2026-07-18 13:42:36.341431572 +0000 UTC m=+606.890996318" Jul 18 13:42:37.011056 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.011019 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm"] Jul 18 13:42:37.013483 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.013466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:37.016207 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.016187 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-562kn\"" Jul 18 13:42:37.024227 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.024203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm"] Jul 18 13:42:37.065943 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.065906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njdc\" (UniqueName: \"kubernetes.io/projected/d2994a17-3a4b-402f-9d08-8a420276497e-kube-api-access-6njdc\") pod \"authorino-operator-75cc8d5cb7-6ndkm\" (UID: \"d2994a17-3a4b-402f-9d08-8a420276497e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:37.167256 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.167219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njdc\" (UniqueName: \"kubernetes.io/projected/d2994a17-3a4b-402f-9d08-8a420276497e-kube-api-access-6njdc\") pod \"authorino-operator-75cc8d5cb7-6ndkm\" (UID: \"d2994a17-3a4b-402f-9d08-8a420276497e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:37.175673 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.175648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njdc\" (UniqueName: \"kubernetes.io/projected/d2994a17-3a4b-402f-9d08-8a420276497e-kube-api-access-6njdc\") pod \"authorino-operator-75cc8d5cb7-6ndkm\" (UID: \"d2994a17-3a4b-402f-9d08-8a420276497e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:37.323571 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.323490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:37.455532 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:37.453862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm"] Jul 18 13:42:38.335330 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:38.335291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" event={"ID":"d2994a17-3a4b-402f-9d08-8a420276497e","Type":"ContainerStarted","Data":"a8e2e3214d50701f05393c84405cdd5465e9963228801f2a9f0c267ffa1a4a16"} Jul 18 13:42:40.347619 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.347582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" event={"ID":"d2994a17-3a4b-402f-9d08-8a420276497e","Type":"ContainerStarted","Data":"846340acb1b38abff790a275d25349eed99b1f3f7e81cc29c38203eab302d2e8"} Jul 18 13:42:40.348248 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.347696 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:40.362433 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.362396 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh"] Jul 18 13:42:40.364688 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.364641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:42:40.367410 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.367391 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rpg8b\"" Jul 18 13:42:40.374822 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.374782 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" podStartSLOduration=2.2521661 podStartE2EDuration="4.374768383s" totalImagesPullingTime="2.122602283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:40.372537539 +0000 UTC m=+610.922102221" watchObservedRunningTime="2026-07-18 13:42:40.374768383 +0000 UTC m=+610.924333062" Jul 18 13:42:40.377853 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.377828 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh"] Jul 18 13:42:40.395059 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.395033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ht68\" (UniqueName: \"kubernetes.io/projected/52affb38-9688-4735-93ee-224fcc863bfb-kube-api-access-9ht68\") pod \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" (UID: \"52affb38-9688-4735-93ee-224fcc863bfb\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:42:40.496473 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.496433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ht68\" (UniqueName: \"kubernetes.io/projected/52affb38-9688-4735-93ee-224fcc863bfb-kube-api-access-9ht68\") pod \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" (UID: \"52affb38-9688-4735-93ee-224fcc863bfb\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:42:40.512520 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.512494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ht68\" (UniqueName: \"kubernetes.io/projected/52affb38-9688-4735-93ee-224fcc863bfb-kube-api-access-9ht68\") pod \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" (UID: \"52affb38-9688-4735-93ee-224fcc863bfb\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:42:40.676175 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.676088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:42:40.811895 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:40.811870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh"] Jul 18 13:42:40.813242 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:40.813215 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52affb38_9688_4735_93ee_224fcc863bfb.slice/crio-0b626349d34e5b74be3ac02678de32d91476c76eb75755ce2d4d1f6c77f730af WatchSource:0}: Error finding container 0b626349d34e5b74be3ac02678de32d91476c76eb75755ce2d4d1f6c77f730af: Status 404 returned error can't find the container with id 0b626349d34e5b74be3ac02678de32d91476c76eb75755ce2d4d1f6c77f730af Jul 18 13:42:41.354437 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:41.354402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" event={"ID":"52affb38-9688-4735-93ee-224fcc863bfb","Type":"ContainerStarted","Data":"0b626349d34e5b74be3ac02678de32d91476c76eb75755ce2d4d1f6c77f730af"} Jul 18 13:42:41.972910 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:41.972883 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:41.973095 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:41.972932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:41.973095 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:41.972982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:41.976987 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:41.976962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd8c85fc8-2kwrs" Jul 18 13:42:42.041032 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:42.040995 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7847b7df5-7fp9r"] Jul 18 13:42:43.367080 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:43.367048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" event={"ID":"52affb38-9688-4735-93ee-224fcc863bfb","Type":"ContainerStarted","Data":"78506484d349d3624d3036bcbc06897dcdfd29c3d506858f78e6f80314bba974"} Jul 18 13:42:43.367507 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:43.367105 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:42:43.387709 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:43.387659 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" podStartSLOduration=1.799897451 podStartE2EDuration="3.38764165s" totalImagesPullingTime="1.587744199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:43.386462986 +0000 UTC m=+613.936027666" watchObservedRunningTime="2026-07-18 13:42:43.38764165 +0000 UTC m=+613.937206344" Jul 18 13:42:44.758724 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.758683 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j"] Jul 18 13:42:44.762664 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.762647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:44.765445 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.765422 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gfbgs\"" Jul 18 13:42:44.774686 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.774657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j"] Jul 18 13:42:44.834451 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.834370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqfj\" (UniqueName: \"kubernetes.io/projected/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-kube-api-access-vcqfj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:44.834595 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.834504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:44.935782 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.935745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:44.935782 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.935786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqfj\" (UniqueName: \"kubernetes.io/projected/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-kube-api-access-vcqfj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:44.936141 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.936119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:44.945117 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:44.945081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqfj\" (UniqueName: \"kubernetes.io/projected/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-kube-api-access-vcqfj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:45.073838 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:45.073748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:45.210843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:45.210814 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j"] Jul 18 13:42:45.212400 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:45.212362 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8763d13_0d7d_47c1_9062_8470fdcd8b9b.slice/crio-5f3145e0e906424c52ab7da73ad60ab014852f3b862e12c5031e0bf2b28d79d8 WatchSource:0}: Error finding container 5f3145e0e906424c52ab7da73ad60ab014852f3b862e12c5031e0bf2b28d79d8: Status 404 returned error can't find the container with id 5f3145e0e906424c52ab7da73ad60ab014852f3b862e12c5031e0bf2b28d79d8 Jul 18 13:42:45.379129 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:45.379041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" event={"ID":"b8763d13-0d7d-47c1-9062-8470fdcd8b9b","Type":"ContainerStarted","Data":"5f3145e0e906424c52ab7da73ad60ab014852f3b862e12c5031e0bf2b28d79d8"} Jul 18 13:42:47.331054 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:47.331021 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tmnxn" Jul 18 13:42:50.409781 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:50.409744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" event={"ID":"b8763d13-0d7d-47c1-9062-8470fdcd8b9b","Type":"ContainerStarted","Data":"63cec66d1028e7b5189e9b5ea54dcc0c901bd06d60319cba0893f4dee8ad36af"} Jul 18 13:42:50.410188 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:50.409832 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:42:50.431665 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:50.431615 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" podStartSLOduration=2.149131404 podStartE2EDuration="6.431598464s" totalImagesPullingTime="4.28246706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:42:50.428397603 +0000 UTC m=+620.977962276" watchObservedRunningTime="2026-07-18 13:42:50.431598464 +0000 UTC m=+620.981163144" Jul 18 13:42:51.356866 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:51.356836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-6ndkm" Jul 18 13:42:52.351793 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.351755 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62"] Jul 18 13:42:52.356644 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.356625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.359243 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.359217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 18 13:42:52.359347 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.359249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rzx6c\"" Jul 18 13:42:52.359347 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.359294 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 18 13:42:52.366981 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.366957 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62"] Jul 18 13:42:52.402713 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.402684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-kube-api-access-mthgv\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.402858 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.402727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.402858 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.402792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.503810 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.503777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-kube-api-access-mthgv\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.503988 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.503827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.503988 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.503888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.504559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.504536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.506240 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.506217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.512917 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.512894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthgv\" (UniqueName: \"kubernetes.io/projected/4b4d5ac0-f879-47b3-95ca-a6ce4d34d470-kube-api-access-mthgv\") pod \"kuadrant-console-plugin-66b5774985-rvf62\" (UID: \"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.666236 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.666151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" Jul 18 13:42:52.798733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:52.798706 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62"] Jul 18 13:42:52.799936 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:42:52.799904 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b4d5ac0_f879_47b3_95ca_a6ce4d34d470.slice/crio-a3f45f1529e25e4e81d1eef3a702762e68c6cbef67eb87c47077a5eb1909d349 WatchSource:0}: Error finding container a3f45f1529e25e4e81d1eef3a702762e68c6cbef67eb87c47077a5eb1909d349: Status 404 returned error can't find the container with id a3f45f1529e25e4e81d1eef3a702762e68c6cbef67eb87c47077a5eb1909d349 Jul 18 13:42:53.427746 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:53.427708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" event={"ID":"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470","Type":"ContainerStarted","Data":"a3f45f1529e25e4e81d1eef3a702762e68c6cbef67eb87c47077a5eb1909d349"} Jul 18 13:42:54.375277 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:42:54.375246 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:01.417809 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:01.417758 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:03.100207 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.099292 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j"] Jul 18 13:43:03.100207 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.099554 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" containerName="manager" containerID="cri-o://63cec66d1028e7b5189e9b5ea54dcc0c901bd06d60319cba0893f4dee8ad36af" gracePeriod=2 Jul 18 13:43:03.103609 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.103556 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:03.105982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.105951 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j"] Jul 18 13:43:03.121417 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.121339 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh"] Jul 18 13:43:03.121651 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.121601 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" podUID="52affb38-9688-4735-93ee-224fcc863bfb" containerName="manager" containerID="cri-o://78506484d349d3624d3036bcbc06897dcdfd29c3d506858f78e6f80314bba974" gracePeriod=2 Jul 18 13:43:03.124193 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.124146 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:03.124532 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.124505 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq"] Jul 18 13:43:03.125790 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.125767 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" containerName="manager" Jul 18 13:43:03.125915 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.125902 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" containerName="manager" Jul 18 13:43:03.126169 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.126153 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" containerName="manager" Jul 18 13:43:03.143064 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.143021 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"52affb38-9688-4735-93ee-224fcc863bfb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-j7rrh\": pods \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:03.172514 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.172338 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh"] Jul 18 13:43:03.172514 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.172415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq"] Jul 18 13:43:03.172514 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.172450 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk"] Jul 18 13:43:03.172514 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.172467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.173277 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.173244 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52affb38-9688-4735-93ee-224fcc863bfb" containerName="manager" Jul 18 13:43:03.173277 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.173272 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52affb38-9688-4735-93ee-224fcc863bfb" containerName="manager" Jul 18 13:43:03.173495 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.173359 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="52affb38-9688-4735-93ee-224fcc863bfb" containerName="manager" Jul 18 13:43:03.174877 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.174837 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:03.177016 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.176979 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="52affb38-9688-4735-93ee-224fcc863bfb" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:03.178128 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.178060 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk"] Jul 18 13:43:03.178204 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.178164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:03.213982 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.213943 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:03.216007 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:03.215976 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="52affb38-9688-4735-93ee-224fcc863bfb" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:03.308791 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.308763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nchq\" (UniqueName: \"kubernetes.io/projected/1431c65a-db6e-4be3-acb1-fce3df414ec7-kube-api-access-7nchq\") pod \"limitador-operator-controller-manager-859d7b78bd-dscqk\" (UID: \"1431c65a-db6e-4be3-acb1-fce3df414ec7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:03.308960 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.308819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-677rq\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.308960 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.308911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtlb\" (UniqueName: \"kubernetes.io/projected/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-kube-api-access-5wtlb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-677rq\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.409931 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.409834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-677rq\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.409931 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.409901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtlb\" (UniqueName: \"kubernetes.io/projected/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-kube-api-access-5wtlb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-677rq\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.410139 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.409971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nchq\" (UniqueName: \"kubernetes.io/projected/1431c65a-db6e-4be3-acb1-fce3df414ec7-kube-api-access-7nchq\") pod \"limitador-operator-controller-manager-859d7b78bd-dscqk\" (UID: \"1431c65a-db6e-4be3-acb1-fce3df414ec7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:03.410262 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.410240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-677rq\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.418559 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.418506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtlb\" (UniqueName: \"kubernetes.io/projected/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-kube-api-access-5wtlb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-677rq\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.418929 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.418904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nchq\" (UniqueName: \"kubernetes.io/projected/1431c65a-db6e-4be3-acb1-fce3df414ec7-kube-api-access-7nchq\") pod \"limitador-operator-controller-manager-859d7b78bd-dscqk\" (UID: \"1431c65a-db6e-4be3-acb1-fce3df414ec7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:03.598243 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.598205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:03.605429 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:03.605396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:07.066057 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:07.066017 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7847b7df5-7fp9r" podUID="7308cc43-8acb-407b-92b2-0a4d3f98df52" containerName="console" containerID="cri-o://842f8047dc9a66a495abac96f844ef1c2e15cddd666e2dbb78a918139a1873a5" gracePeriod=15 Jul 18 13:43:10.290112 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:10.290069 2564 patch_prober.go:28] interesting pod/console-7847b7df5-7fp9r container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body= Jul 18 13:43:10.290576 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:10.290137 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7847b7df5-7fp9r" podUID="7308cc43-8acb-407b-92b2-0a4d3f98df52" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" Jul 18 13:43:13.557426 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.557235 2564 generic.go:356] "Generic (PLEG): container finished" podID="7308cc43-8acb-407b-92b2-0a4d3f98df52" containerID="842f8047dc9a66a495abac96f844ef1c2e15cddd666e2dbb78a918139a1873a5" exitCode=0 Jul 18 13:43:13.557426 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.557341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847b7df5-7fp9r" event={"ID":"7308cc43-8acb-407b-92b2-0a4d3f98df52","Type":"ContainerDied","Data":"842f8047dc9a66a495abac96f844ef1c2e15cddd666e2dbb78a918139a1873a5"} Jul 18 13:43:13.560160 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.559659 2564 generic.go:356] "Generic (PLEG): container finished" podID="52affb38-9688-4735-93ee-224fcc863bfb" containerID="78506484d349d3624d3036bcbc06897dcdfd29c3d506858f78e6f80314bba974" exitCode=0 Jul 18 13:43:13.562608 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.562547 2564 generic.go:356] "Generic (PLEG): container finished" podID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" containerID="63cec66d1028e7b5189e9b5ea54dcc0c901bd06d60319cba0893f4dee8ad36af" exitCode=0 Jul 18 13:43:13.643519 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.643193 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq"] Jul 18 13:43:13.645734 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:43:13.645701 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5fd64b_b6f0_4a4f_b9bd_dec52ab9d62c.slice/crio-92fef628535931c135ebfd7ef8bb8f3bf860c53df02f248806d9417978b85729 WatchSource:0}: Error finding container 92fef628535931c135ebfd7ef8bb8f3bf860c53df02f248806d9417978b85729: Status 404 returned error can't find the container with id 92fef628535931c135ebfd7ef8bb8f3bf860c53df02f248806d9417978b85729 Jul 18 13:43:13.650530 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.650191 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:43:13.653698 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.653581 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:13.670161 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:13.670058 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:13.674988 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.674964 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:13.675855 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.675833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk"] Jul 18 13:43:13.677122 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:13.677095 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:13.678625 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:43:13.678594 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1431c65a_db6e_4be3_acb1_fce3df414ec7.slice/crio-e93ea5bbcb9e5742cebde10a72d4f56ae09f47f25eaa02ea1e745d54554b5a5b WatchSource:0}: Error finding container e93ea5bbcb9e5742cebde10a72d4f56ae09f47f25eaa02ea1e745d54554b5a5b: Status 404 returned error can't find the container with id e93ea5bbcb9e5742cebde10a72d4f56ae09f47f25eaa02ea1e745d54554b5a5b Jul 18 13:43:13.679105 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:13.679078 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="52affb38-9688-4735-93ee-224fcc863bfb" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:13.816005 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.815975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-config\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816020 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-oauth-config\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816058 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgk6n\" (UniqueName: \"kubernetes.io/projected/7308cc43-8acb-407b-92b2-0a4d3f98df52-kube-api-access-mgk6n\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816086 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-trusted-ca-bundle\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816112 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcqfj\" (UniqueName: \"kubernetes.io/projected/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-kube-api-access-vcqfj\") pod \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " Jul 18 13:43:13.816195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816149 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-oauth-serving-cert\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816187 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-serving-cert\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816513 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-service-ca\") pod \"7308cc43-8acb-407b-92b2-0a4d3f98df52\" (UID: \"7308cc43-8acb-407b-92b2-0a4d3f98df52\") " Jul 18 13:43:13.816513 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816248 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-extensions-socket-volume\") pod \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\" (UID: \"b8763d13-0d7d-47c1-9062-8470fdcd8b9b\") " Jul 18 13:43:13.816513 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816305 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ht68\" (UniqueName: \"kubernetes.io/projected/52affb38-9688-4735-93ee-224fcc863bfb-kube-api-access-9ht68\") pod \"52affb38-9688-4735-93ee-224fcc863bfb\" (UID: \"52affb38-9688-4735-93ee-224fcc863bfb\") " Jul 18 13:43:13.816668 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816428 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-config" (OuterVolumeSpecName: "console-config") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:43:13.816668 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816612 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:43:13.816779 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816692 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-oauth-serving-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.816779 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816709 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.816779 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.816716 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-service-ca" (OuterVolumeSpecName: "service-ca") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:43:13.817105 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.817078 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:43:13.817337 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.817311 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8763d13-0d7d-47c1-9062-8470fdcd8b9b" (UID: "b8763d13-0d7d-47c1-9062-8470fdcd8b9b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:43:13.818478 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.818447 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-kube-api-access-vcqfj" (OuterVolumeSpecName: "kube-api-access-vcqfj") pod "b8763d13-0d7d-47c1-9062-8470fdcd8b9b" (UID: "b8763d13-0d7d-47c1-9062-8470fdcd8b9b"). InnerVolumeSpecName "kube-api-access-vcqfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:43:13.818692 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.818672 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7308cc43-8acb-407b-92b2-0a4d3f98df52-kube-api-access-mgk6n" (OuterVolumeSpecName: "kube-api-access-mgk6n") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "kube-api-access-mgk6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:43:13.818761 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.818715 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52affb38-9688-4735-93ee-224fcc863bfb-kube-api-access-9ht68" (OuterVolumeSpecName: "kube-api-access-9ht68") pod "52affb38-9688-4735-93ee-224fcc863bfb" (UID: "52affb38-9688-4735-93ee-224fcc863bfb"). InnerVolumeSpecName "kube-api-access-9ht68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:43:13.818803 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.818761 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:43:13.818843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.818828 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7308cc43-8acb-407b-92b2-0a4d3f98df52" (UID: "7308cc43-8acb-407b-92b2-0a4d3f98df52"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:43:13.917695 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917665 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-oauth-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917695 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917692 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgk6n\" (UniqueName: \"kubernetes.io/projected/7308cc43-8acb-407b-92b2-0a4d3f98df52-kube-api-access-mgk6n\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917695 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917703 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-trusted-ca-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917713 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcqfj\" (UniqueName: \"kubernetes.io/projected/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-kube-api-access-vcqfj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917723 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7308cc43-8acb-407b-92b2-0a4d3f98df52-console-serving-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917732 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7308cc43-8acb-407b-92b2-0a4d3f98df52-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917740 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8763d13-0d7d-47c1-9062-8470fdcd8b9b-extensions-socket-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.917914 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.917750 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ht68\" (UniqueName: \"kubernetes.io/projected/52affb38-9688-4735-93ee-224fcc863bfb-kube-api-access-9ht68\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:13.973676 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.973589 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52affb38-9688-4735-93ee-224fcc863bfb" path="/var/lib/kubelet/pods/52affb38-9688-4735-93ee-224fcc863bfb/volumes" Jul 18 13:43:13.973997 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:13.973981 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" path="/var/lib/kubelet/pods/b8763d13-0d7d-47c1-9062-8470fdcd8b9b/volumes" Jul 18 13:43:14.574633 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.574589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" event={"ID":"4b4d5ac0-f879-47b3-95ca-a6ce4d34d470","Type":"ContainerStarted","Data":"439434ae23f76a84985c1dbef0a50e0589be5b8d7f4d2379cc236514f0b3b22c"} Jul 18 13:43:14.575993 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.575969 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7847b7df5-7fp9r" Jul 18 13:43:14.576111 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.575967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7847b7df5-7fp9r" event={"ID":"7308cc43-8acb-407b-92b2-0a4d3f98df52","Type":"ContainerDied","Data":"30b7994478cda83ba21c5bf2a49976ecff4f502602236d877165d60a1d7705f6"} Jul 18 13:43:14.576152 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.576135 2564 scope.go:117] "RemoveContainer" containerID="842f8047dc9a66a495abac96f844ef1c2e15cddd666e2dbb78a918139a1873a5" Jul 18 13:43:14.577419 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.577248 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:14.578961 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.578945 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:14.580982 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.580960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" event={"ID":"1431c65a-db6e-4be3-acb1-fce3df414ec7","Type":"ContainerStarted","Data":"563eb0e852fee0659ee81eff28e0098092583624d8f56014ed47f970d74a8300"} Jul 18 13:43:14.581068 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.581000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" event={"ID":"1431c65a-db6e-4be3-acb1-fce3df414ec7","Type":"ContainerStarted","Data":"e93ea5bbcb9e5742cebde10a72d4f56ae09f47f25eaa02ea1e745d54554b5a5b"} Jul 18 13:43:14.581068 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.581028 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:14.582701 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.582681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" event={"ID":"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c","Type":"ContainerStarted","Data":"5118a80ff65dffa2598e6c9a2493c82caedf49bcb800b4edf9be97335c619a1f"} Jul 18 13:43:14.582807 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.582722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" event={"ID":"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c","Type":"ContainerStarted","Data":"92fef628535931c135ebfd7ef8bb8f3bf860c53df02f248806d9417978b85729"} Jul 18 13:43:14.582807 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.582750 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:14.586072 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.586057 2564 scope.go:117] "RemoveContainer" containerID="78506484d349d3624d3036bcbc06897dcdfd29c3d506858f78e6f80314bba974" Jul 18 13:43:14.592666 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.592618 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvf62" podStartSLOduration=1.844391492 podStartE2EDuration="22.592601843s" totalImagesPullingTime="20.748210351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:43:14.591817784 +0000 UTC m=+645.141382477" watchObservedRunningTime="2026-07-18 13:43:14.592601843 +0000 UTC m=+645.142166525" Jul 18 13:43:14.594774 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.594756 2564 scope.go:117] "RemoveContainer" containerID="63cec66d1028e7b5189e9b5ea54dcc0c901bd06d60319cba0893f4dee8ad36af" Jul 18 13:43:14.609652 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.609609 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" podStartSLOduration=11.609588702 podStartE2EDuration="11.609588702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:43:14.608545169 +0000 UTC m=+645.158109850" watchObservedRunningTime="2026-07-18 13:43:14.609588702 +0000 UTC m=+645.159153384" Jul 18 13:43:14.626465 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:14.626424 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7gd5j\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b8763d13-0d7d-47c1-9062-8470fdcd8b9b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7gd5j" Jul 18 13:43:14.626655 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.626563 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" podStartSLOduration=11.626549005 podStartE2EDuration="11.626549005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:43:14.624301758 +0000 UTC m=+645.173866441" watchObservedRunningTime="2026-07-18 13:43:14.626549005 +0000 UTC m=+645.176113686" Jul 18 13:43:14.628509 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:43:14.628476 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-j7rrh\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="52affb38-9688-4735-93ee-224fcc863bfb" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7rrh" Jul 18 13:43:14.642179 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.642149 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7847b7df5-7fp9r"] Jul 18 13:43:14.647770 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:14.647747 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7847b7df5-7fp9r"] Jul 18 13:43:15.971942 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:15.971909 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7308cc43-8acb-407b-92b2-0a4d3f98df52" path="/var/lib/kubelet/pods/7308cc43-8acb-407b-92b2-0a4d3f98df52/volumes" Jul 18 13:43:18.459689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.459659 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq"] Jul 18 13:43:18.460106 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.459867 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" podUID="fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" containerName="manager" containerID="cri-o://5118a80ff65dffa2598e6c9a2493c82caedf49bcb800b4edf9be97335c619a1f" gracePeriod=10 Jul 18 13:43:18.461828 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.461804 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:18.608212 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.608180 2564 generic.go:356] "Generic (PLEG): container finished" podID="fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" containerID="5118a80ff65dffa2598e6c9a2493c82caedf49bcb800b4edf9be97335c619a1f" exitCode=0 Jul 18 13:43:18.608425 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.608253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" event={"ID":"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c","Type":"ContainerDied","Data":"5118a80ff65dffa2598e6c9a2493c82caedf49bcb800b4edf9be97335c619a1f"} Jul 18 13:43:18.706434 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.706405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:18.753784 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.753688 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wtlb\" (UniqueName: \"kubernetes.io/projected/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-kube-api-access-5wtlb\") pod \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " Jul 18 13:43:18.753784 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.753723 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-extensions-socket-volume\") pod \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\" (UID: \"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c\") " Jul 18 13:43:18.754134 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.754110 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" (UID: "fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 13:43:18.755880 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.755847 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-kube-api-access-5wtlb" (OuterVolumeSpecName: "kube-api-access-5wtlb") pod "fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" (UID: "fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c"). InnerVolumeSpecName "kube-api-access-5wtlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:43:18.854488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.854435 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-extensions-socket-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:18.854488 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:18.854481 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wtlb\" (UniqueName: \"kubernetes.io/projected/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c-kube-api-access-5wtlb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:43:19.615509 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:19.615479 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" Jul 18 13:43:19.615965 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:19.615473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq" event={"ID":"fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c","Type":"ContainerDied","Data":"92fef628535931c135ebfd7ef8bb8f3bf860c53df02f248806d9417978b85729"} Jul 18 13:43:19.615965 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:19.615625 2564 scope.go:117] "RemoveContainer" containerID="5118a80ff65dffa2598e6c9a2493c82caedf49bcb800b4edf9be97335c619a1f" Jul 18 13:43:19.644137 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:19.644107 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq"] Jul 18 13:43:19.656032 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:19.651528 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-677rq"] Jul 18 13:43:19.973172 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:19.973138 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" path="/var/lib/kubelet/pods/fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c/volumes" Jul 18 13:43:25.593142 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:25.593112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dscqk" Jul 18 13:43:34.708111 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708072 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579"] Jul 18 13:43:34.708692 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708670 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7308cc43-8acb-407b-92b2-0a4d3f98df52" containerName="console" Jul 18 13:43:34.708738 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708696 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7308cc43-8acb-407b-92b2-0a4d3f98df52" containerName="console" Jul 18 13:43:34.708738 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708713 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" containerName="manager" Jul 18 13:43:34.708738 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708718 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" containerName="manager" Jul 18 13:43:34.708834 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708802 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7308cc43-8acb-407b-92b2-0a4d3f98df52" containerName="console" Jul 18 13:43:34.708834 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.708813 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5fd64b-b6f0-4a4f-b9bd-dec52ab9d62c" containerName="manager" Jul 18 13:43:34.902588 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.902543 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579"] Jul 18 13:43:34.902785 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.902681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.905291 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.905265 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 13:43:34.905451 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.905428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x8494\"" Jul 18 13:43:34.983266 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983266 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983266 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983594 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983663 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983722 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983774 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983857 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:34.983922 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:34.983876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwvb\" (UniqueName: \"kubernetes.io/projected/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-kube-api-access-7nwvb\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084601 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084601 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwvb\" (UniqueName: \"kubernetes.io/projected/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-kube-api-access-7nwvb\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.084840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085102 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085102 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.084961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085102 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.085058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085256 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.085116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085256 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.085147 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085442 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.085404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.085496 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.085455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.087073 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.087050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.087283 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.087265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.093689 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.093663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.093918 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.093735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwvb\" (UniqueName: \"kubernetes.io/projected/0c4a5666-4fc5-4262-9cc9-56679b6d3c39-kube-api-access-7nwvb\") pod \"maas-default-gateway-openshift-default-56654dcf7-bk579\" (UID: \"0c4a5666-4fc5-4262-9cc9-56679b6d3c39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.213281 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.213246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:35.408430 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.408401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579"] Jul 18 13:43:35.410298 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:43:35.410264 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4a5666_4fc5_4262_9cc9_56679b6d3c39.slice/crio-7f960eec0f84cb97b59da4151fe4800cd9e295b5515e2d144fdf78bebb75a09d WatchSource:0}: Error finding container 7f960eec0f84cb97b59da4151fe4800cd9e295b5515e2d144fdf78bebb75a09d: Status 404 returned error can't find the container with id 7f960eec0f84cb97b59da4151fe4800cd9e295b5515e2d144fdf78bebb75a09d Jul 18 13:43:35.699915 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:35.699882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" event={"ID":"0c4a5666-4fc5-4262-9cc9-56679b6d3c39","Type":"ContainerStarted","Data":"7f960eec0f84cb97b59da4151fe4800cd9e295b5515e2d144fdf78bebb75a09d"} Jul 18 13:43:38.687138 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:38.687091 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:43:38.687470 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:38.687175 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:43:38.687470 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:38.687208 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:43:39.724590 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:39.724547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" event={"ID":"0c4a5666-4fc5-4262-9cc9-56679b6d3c39","Type":"ContainerStarted","Data":"7bb6d447282ffb6c5e72ccf147e4107adc79af42c7495dab60e66d151902ff77"} Jul 18 13:43:39.745814 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:39.745753 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" podStartSLOduration=2.471465441 podStartE2EDuration="5.745733119s" totalImagesPullingTime="3.274267678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:43:39.745184578 +0000 UTC m=+670.294749258" watchObservedRunningTime="2026-07-18 13:43:39.745733119 +0000 UTC m=+670.295297801" Jul 18 13:43:40.213937 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:40.213908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:40.218708 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:40.218681 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:40.729691 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:40.729663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:40.730696 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:40.730674 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:43:49.083046 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.083007 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:43:49.151642 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.151613 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:43:49.151816 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.151771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.154266 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.154234 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 18 13:43:49.182574 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.182540 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:43:49.217123 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.217079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56e6463f-6883-4d33-9c92-cefc04e264d7-config-file\") pod \"limitador-limitador-64c94fd64f-8dt6k\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.217290 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.217137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4kg\" (UniqueName: \"kubernetes.io/projected/56e6463f-6883-4d33-9c92-cefc04e264d7-kube-api-access-cw4kg\") pod \"limitador-limitador-64c94fd64f-8dt6k\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.318511 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.318475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4kg\" (UniqueName: \"kubernetes.io/projected/56e6463f-6883-4d33-9c92-cefc04e264d7-kube-api-access-cw4kg\") pod \"limitador-limitador-64c94fd64f-8dt6k\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.318703 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.318590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56e6463f-6883-4d33-9c92-cefc04e264d7-config-file\") pod \"limitador-limitador-64c94fd64f-8dt6k\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.319205 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.319179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56e6463f-6883-4d33-9c92-cefc04e264d7-config-file\") pod \"limitador-limitador-64c94fd64f-8dt6k\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.326349 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.326321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4kg\" (UniqueName: \"kubernetes.io/projected/56e6463f-6883-4d33-9c92-cefc04e264d7-kube-api-access-cw4kg\") pod \"limitador-limitador-64c94fd64f-8dt6k\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.467242 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.467203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:49.486826 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.486795 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:43:49.491566 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.491544 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.503570 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.503528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:43:49.516357 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.516320 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:43:49.609733 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.609694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:43:49.614984 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:43:49.614946 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e6463f_6883_4d33_9c92_cefc04e264d7.slice/crio-3f5afc5d20dfbe62b9b4d91e429da40dfc77b4f71992e0923bbe51e347d93130 WatchSource:0}: Error finding container 3f5afc5d20dfbe62b9b4d91e429da40dfc77b4f71992e0923bbe51e347d93130: Status 404 returned error can't find the container with id 3f5afc5d20dfbe62b9b4d91e429da40dfc77b4f71992e0923bbe51e347d93130 Jul 18 13:43:49.620659 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.620635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f69186e-c708-4b22-8fa8-4f0e622ece75-config-file\") pod \"limitador-limitador-54d6d988d5-tw58w\" (UID: \"7f69186e-c708-4b22-8fa8-4f0e622ece75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.620777 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.620711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdmh\" (UniqueName: \"kubernetes.io/projected/7f69186e-c708-4b22-8fa8-4f0e622ece75-kube-api-access-vtdmh\") pod \"limitador-limitador-54d6d988d5-tw58w\" (UID: \"7f69186e-c708-4b22-8fa8-4f0e622ece75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.721726 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.721630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f69186e-c708-4b22-8fa8-4f0e622ece75-config-file\") pod \"limitador-limitador-54d6d988d5-tw58w\" (UID: \"7f69186e-c708-4b22-8fa8-4f0e622ece75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.721882 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.721795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdmh\" (UniqueName: \"kubernetes.io/projected/7f69186e-c708-4b22-8fa8-4f0e622ece75-kube-api-access-vtdmh\") pod \"limitador-limitador-54d6d988d5-tw58w\" (UID: \"7f69186e-c708-4b22-8fa8-4f0e622ece75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.722312 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.722286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f69186e-c708-4b22-8fa8-4f0e622ece75-config-file\") pod \"limitador-limitador-54d6d988d5-tw58w\" (UID: \"7f69186e-c708-4b22-8fa8-4f0e622ece75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.729786 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.729756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdmh\" (UniqueName: \"kubernetes.io/projected/7f69186e-c708-4b22-8fa8-4f0e622ece75-kube-api-access-vtdmh\") pod \"limitador-limitador-54d6d988d5-tw58w\" (UID: \"7f69186e-c708-4b22-8fa8-4f0e622ece75\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.781018 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.780982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" event={"ID":"56e6463f-6883-4d33-9c92-cefc04e264d7","Type":"ContainerStarted","Data":"3f5afc5d20dfbe62b9b4d91e429da40dfc77b4f71992e0923bbe51e347d93130"} Jul 18 13:43:49.803553 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.803527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:49.937511 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:49.937488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:43:49.939360 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:43:49.939334 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f69186e_c708_4b22_8fa8_4f0e622ece75.slice/crio-4d37cf724d6b42bb9a6c7bb9e021dbdce346e7bbe6e746a4e2b9d9ea7e815d50 WatchSource:0}: Error finding container 4d37cf724d6b42bb9a6c7bb9e021dbdce346e7bbe6e746a4e2b9d9ea7e815d50: Status 404 returned error can't find the container with id 4d37cf724d6b42bb9a6c7bb9e021dbdce346e7bbe6e746a4e2b9d9ea7e815d50 Jul 18 13:43:50.802093 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:50.802004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" event={"ID":"7f69186e-c708-4b22-8fa8-4f0e622ece75","Type":"ContainerStarted","Data":"4d37cf724d6b42bb9a6c7bb9e021dbdce346e7bbe6e746a4e2b9d9ea7e815d50"} Jul 18 13:43:53.822139 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:53.822100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" event={"ID":"56e6463f-6883-4d33-9c92-cefc04e264d7","Type":"ContainerStarted","Data":"c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d"} Jul 18 13:43:53.822631 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:53.822243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:43:53.823921 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:53.823902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" event={"ID":"7f69186e-c708-4b22-8fa8-4f0e622ece75","Type":"ContainerStarted","Data":"a1e38666d0799de51af83e837452b43296aca04e43ee1f10b078bc9b485a86dc"} Jul 18 13:43:53.823996 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:53.823971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:43:53.839462 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:53.839413 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" podStartSLOduration=1.666062421 podStartE2EDuration="4.839394818s" totalImagesPullingTime="3.173332397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:43:53.836502883 +0000 UTC m=+684.386067564" watchObservedRunningTime="2026-07-18 13:43:53.839394818 +0000 UTC m=+684.388959499" Jul 18 13:43:53.857889 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:43:53.857839 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" podStartSLOduration=2.017041498 podStartE2EDuration="4.857824639s" totalImagesPullingTime="2.840783141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:43:53.855952643 +0000 UTC m=+684.405517353" watchObservedRunningTime="2026-07-18 13:43:53.857824639 +0000 UTC m=+684.407389356" Jul 18 13:44:04.830477 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:04.830446 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:44:04.830852 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:04.830516 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tw58w" Jul 18 13:44:04.888621 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:04.888589 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:44:04.888783 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:04.888759 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" podUID="56e6463f-6883-4d33-9c92-cefc04e264d7" containerName="limitador" containerID="cri-o://c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d" gracePeriod=30 Jul 18 13:44:05.448678 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.448613 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:44:05.564519 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.564485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56e6463f-6883-4d33-9c92-cefc04e264d7-config-file\") pod \"56e6463f-6883-4d33-9c92-cefc04e264d7\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " Jul 18 13:44:05.564708 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.564566 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw4kg\" (UniqueName: \"kubernetes.io/projected/56e6463f-6883-4d33-9c92-cefc04e264d7-kube-api-access-cw4kg\") pod \"56e6463f-6883-4d33-9c92-cefc04e264d7\" (UID: \"56e6463f-6883-4d33-9c92-cefc04e264d7\") " Jul 18 13:44:05.565108 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.565060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e6463f-6883-4d33-9c92-cefc04e264d7-config-file" (OuterVolumeSpecName: "config-file") pod "56e6463f-6883-4d33-9c92-cefc04e264d7" (UID: "56e6463f-6883-4d33-9c92-cefc04e264d7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 13:44:05.571435 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.567718 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e6463f-6883-4d33-9c92-cefc04e264d7-kube-api-access-cw4kg" (OuterVolumeSpecName: "kube-api-access-cw4kg") pod "56e6463f-6883-4d33-9c92-cefc04e264d7" (UID: "56e6463f-6883-4d33-9c92-cefc04e264d7"). InnerVolumeSpecName "kube-api-access-cw4kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:44:05.571435 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.567768 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56e6463f-6883-4d33-9c92-cefc04e264d7-config-file\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:44:05.668645 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.668604 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw4kg\" (UniqueName: \"kubernetes.io/projected/56e6463f-6883-4d33-9c92-cefc04e264d7-kube-api-access-cw4kg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:44:05.886406 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.886274 2564 generic.go:356] "Generic (PLEG): container finished" podID="56e6463f-6883-4d33-9c92-cefc04e264d7" containerID="c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d" exitCode=0 Jul 18 13:44:05.886406 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.886339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" event={"ID":"56e6463f-6883-4d33-9c92-cefc04e264d7","Type":"ContainerDied","Data":"c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d"} Jul 18 13:44:05.886406 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.886351 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" Jul 18 13:44:05.886942 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.886466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8dt6k" event={"ID":"56e6463f-6883-4d33-9c92-cefc04e264d7","Type":"ContainerDied","Data":"3f5afc5d20dfbe62b9b4d91e429da40dfc77b4f71992e0923bbe51e347d93130"} Jul 18 13:44:05.886942 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.886503 2564 scope.go:117] "RemoveContainer" containerID="c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d" Jul 18 13:44:05.895959 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.895798 2564 scope.go:117] "RemoveContainer" containerID="c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d" Jul 18 13:44:05.896074 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:44:05.896055 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d\": container with ID starting with c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d not found: ID does not exist" containerID="c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d" Jul 18 13:44:05.896116 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.896085 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d"} err="failed to get container status \"c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d\": rpc error: code = NotFound desc = could not find container \"c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d\": container with ID starting with c31ab3a3505fd8104eb05371edd0e26e9955791271aef37a09e0e68faa33ee8d not found: ID does not exist" Jul 18 13:44:05.908394 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.908349 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:44:05.912537 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.912506 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8dt6k"] Jul 18 13:44:05.972387 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:05.972334 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e6463f-6883-4d33-9c92-cefc04e264d7" path="/var/lib/kubelet/pods/56e6463f-6883-4d33-9c92-cefc04e264d7/volumes" Jul 18 13:44:26.352084 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.351998 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f744f9d59-2np42"] Jul 18 13:44:26.352594 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.352514 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e6463f-6883-4d33-9c92-cefc04e264d7" containerName="limitador" Jul 18 13:44:26.352594 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.352534 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e6463f-6883-4d33-9c92-cefc04e264d7" containerName="limitador" Jul 18 13:44:26.352681 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.352605 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e6463f-6883-4d33-9c92-cefc04e264d7" containerName="limitador" Jul 18 13:44:26.355506 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.355489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:26.357837 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.357811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 13:44:26.358050 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.358033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xbm2b\"" Jul 18 13:44:26.358986 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.358965 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 13:44:26.364623 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.364604 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f744f9d59-2np42"] Jul 18 13:44:26.438490 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.438446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:26.438667 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.438525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9xc\" (UniqueName: \"kubernetes.io/projected/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-kube-api-access-bz9xc\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:26.539357 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.539318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9xc\" (UniqueName: \"kubernetes.io/projected/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-kube-api-access-bz9xc\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:26.539595 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.539420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:26.539595 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:44:26.539506 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 18 13:44:26.539595 ip-10-0-136-156 kubenswrapper[2564]: E0718 13:44:26.539566 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert podName:ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9 nodeName:}" failed. No retries permitted until 2026-07-18 13:44:27.039551011 +0000 UTC m=+717.589115670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert") pod "maas-controller-5f744f9d59-2np42" (UID: "ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9") : secret "maas-controller-webhook-cert" not found Jul 18 13:44:26.547812 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:26.547780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9xc\" (UniqueName: \"kubernetes.io/projected/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-kube-api-access-bz9xc\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:27.043994 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:27.043955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:27.046411 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:27.046358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert\") pod \"maas-controller-5f744f9d59-2np42\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:27.267807 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:27.267772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:27.402978 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:27.402945 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f744f9d59-2np42"] Jul 18 13:44:27.404384 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:44:27.404351 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf3a2dc_6a3c_45c4_91cf_ece14ef0d6f9.slice/crio-4744571df6a672fb231a387f792f72ef18aa4500375e780f99ead540c2c174b1 WatchSource:0}: Error finding container 4744571df6a672fb231a387f792f72ef18aa4500375e780f99ead540c2c174b1: Status 404 returned error can't find the container with id 4744571df6a672fb231a387f792f72ef18aa4500375e780f99ead540c2c174b1 Jul 18 13:44:27.997833 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:27.997753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f744f9d59-2np42" event={"ID":"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9","Type":"ContainerStarted","Data":"4744571df6a672fb231a387f792f72ef18aa4500375e780f99ead540c2c174b1"} Jul 18 13:44:30.010355 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:30.010318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f744f9d59-2np42" event={"ID":"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9","Type":"ContainerStarted","Data":"7515d7652223e50c7e5b8913e1f7550bb242a390300032e4abdaf9017a3184e3"} Jul 18 13:44:30.010756 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:30.010410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:44:30.028800 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:30.028752 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f744f9d59-2np42" podStartSLOduration=1.8851683270000001 podStartE2EDuration="4.028736395s" totalImagesPullingTime="2.143568068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:44:30.02719086 +0000 UTC m=+720.576755565" watchObservedRunningTime="2026-07-18 13:44:30.028736395 +0000 UTC m=+720.578301076" Jul 18 13:44:41.018440 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:44:41.018410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:45:00.604604 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.604564 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-689d469879-6d7n7"] Jul 18 13:45:00.612952 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.612922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.615951 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.615927 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 18 13:45:00.616449 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.616426 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 18 13:45:00.616773 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.616443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-97gtb\"" Jul 18 13:45:00.616897 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.616513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 13:45:00.618126 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.618105 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-689d469879-6d7n7"] Jul 18 13:45:00.746991 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.746958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9qx\" (UniqueName: \"kubernetes.io/projected/0097e143-75f2-4a20-bbcc-09c29df116c3-kube-api-access-zq9qx\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.747170 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.747026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0097e143-75f2-4a20-bbcc-09c29df116c3-maas-api-tls\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.747170 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.747071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0097e143-75f2-4a20-bbcc-09c29df116c3-openshift-service-ca\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.847714 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.847673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0097e143-75f2-4a20-bbcc-09c29df116c3-openshift-service-ca\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.847888 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.847776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9qx\" (UniqueName: \"kubernetes.io/projected/0097e143-75f2-4a20-bbcc-09c29df116c3-kube-api-access-zq9qx\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.847888 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.847843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0097e143-75f2-4a20-bbcc-09c29df116c3-maas-api-tls\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.848313 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.848291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0097e143-75f2-4a20-bbcc-09c29df116c3-openshift-service-ca\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.850280 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.850254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0097e143-75f2-4a20-bbcc-09c29df116c3-maas-api-tls\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.860291 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.860230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9qx\" (UniqueName: \"kubernetes.io/projected/0097e143-75f2-4a20-bbcc-09c29df116c3-kube-api-access-zq9qx\") pod \"maas-api-689d469879-6d7n7\" (UID: \"0097e143-75f2-4a20-bbcc-09c29df116c3\") " pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:00.925096 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:00.925064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:01.080711 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.080678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-689d469879-6d7n7"] Jul 18 13:45:01.082570 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:45:01.082544 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0097e143_75f2_4a20_bbcc_09c29df116c3.slice/crio-cbd93fa22932c9f3f6bb7669e145b7c590e8488787e0637808a305a9efce2ec4 WatchSource:0}: Error finding container cbd93fa22932c9f3f6bb7669e145b7c590e8488787e0637808a305a9efce2ec4: Status 404 returned error can't find the container with id cbd93fa22932c9f3f6bb7669e145b7c590e8488787e0637808a305a9efce2ec4 Jul 18 13:45:01.084213 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.084190 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 13:45:01.107236 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.107206 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-frbgv"] Jul 18 13:45:01.112672 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.112613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.115449 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.115428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wxf2l\"" Jul 18 13:45:01.115570 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.115468 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 18 13:45:01.122261 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.122235 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-frbgv"] Jul 18 13:45:01.153561 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.153528 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-kpt67"] Jul 18 13:45:01.163509 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.163481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:45:01.163682 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.163603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.167786 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.167757 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-kpt67"] Jul 18 13:45:01.175843 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.175807 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" event={"ID":"0097e143-75f2-4a20-bbcc-09c29df116c3","Type":"ContainerStarted","Data":"cbd93fa22932c9f3f6bb7669e145b7c590e8488787e0637808a305a9efce2ec4"} Jul 18 13:45:01.251695 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.251656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852dde2-e6ec-4b49-a844-d58db0e9a726-plugins-config-volume\") pod \"payload-processing-5bf47cd988-kpt67\" (UID: \"5852dde2-e6ec-4b49-a844-d58db0e9a726\") " pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.251869 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.251709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2765f9f-2e3a-4810-a3e3-52afab8057ed-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-frbgv\" (UID: \"e2765f9f-2e3a-4810-a3e3-52afab8057ed\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.251869 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.251781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56gk\" (UniqueName: \"kubernetes.io/projected/e2765f9f-2e3a-4810-a3e3-52afab8057ed-kube-api-access-n56gk\") pod \"payload-pre-processing-7677b774bb-frbgv\" (UID: \"e2765f9f-2e3a-4810-a3e3-52afab8057ed\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.251869 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.251852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8qb\" (UniqueName: \"kubernetes.io/projected/5852dde2-e6ec-4b49-a844-d58db0e9a726-kube-api-access-fz8qb\") pod \"payload-processing-5bf47cd988-kpt67\" (UID: \"5852dde2-e6ec-4b49-a844-d58db0e9a726\") " pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.352860 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.352815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852dde2-e6ec-4b49-a844-d58db0e9a726-plugins-config-volume\") pod \"payload-processing-5bf47cd988-kpt67\" (UID: \"5852dde2-e6ec-4b49-a844-d58db0e9a726\") " pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.353050 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.352873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2765f9f-2e3a-4810-a3e3-52afab8057ed-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-frbgv\" (UID: \"e2765f9f-2e3a-4810-a3e3-52afab8057ed\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.353050 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.352933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56gk\" (UniqueName: \"kubernetes.io/projected/e2765f9f-2e3a-4810-a3e3-52afab8057ed-kube-api-access-n56gk\") pod \"payload-pre-processing-7677b774bb-frbgv\" (UID: \"e2765f9f-2e3a-4810-a3e3-52afab8057ed\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.353050 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.352996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8qb\" (UniqueName: \"kubernetes.io/projected/5852dde2-e6ec-4b49-a844-d58db0e9a726-kube-api-access-fz8qb\") pod \"payload-processing-5bf47cd988-kpt67\" (UID: \"5852dde2-e6ec-4b49-a844-d58db0e9a726\") " pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.353563 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.353541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852dde2-e6ec-4b49-a844-d58db0e9a726-plugins-config-volume\") pod \"payload-processing-5bf47cd988-kpt67\" (UID: \"5852dde2-e6ec-4b49-a844-d58db0e9a726\") " pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.353647 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.353629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2765f9f-2e3a-4810-a3e3-52afab8057ed-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-frbgv\" (UID: \"e2765f9f-2e3a-4810-a3e3-52afab8057ed\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.364103 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.364032 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8qb\" (UniqueName: \"kubernetes.io/projected/5852dde2-e6ec-4b49-a844-d58db0e9a726-kube-api-access-fz8qb\") pod \"payload-processing-5bf47cd988-kpt67\" (UID: \"5852dde2-e6ec-4b49-a844-d58db0e9a726\") " pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.364103 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.364037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56gk\" (UniqueName: \"kubernetes.io/projected/e2765f9f-2e3a-4810-a3e3-52afab8057ed-kube-api-access-n56gk\") pod \"payload-pre-processing-7677b774bb-frbgv\" (UID: \"e2765f9f-2e3a-4810-a3e3-52afab8057ed\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.425085 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.425034 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:01.478359 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.478156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:01.574214 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.574169 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-frbgv"] Jul 18 13:45:01.576720 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:45:01.576687 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2765f9f_2e3a_4810_a3e3_52afab8057ed.slice/crio-ece382e0f26b9e3a35c417e0d1e7cbceadd84a316c6dd056dc95a23e5fd56e13 WatchSource:0}: Error finding container ece382e0f26b9e3a35c417e0d1e7cbceadd84a316c6dd056dc95a23e5fd56e13: Status 404 returned error can't find the container with id ece382e0f26b9e3a35c417e0d1e7cbceadd84a316c6dd056dc95a23e5fd56e13 Jul 18 13:45:01.659192 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:01.659163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-kpt67"] Jul 18 13:45:01.661122 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:45:01.661087 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5852dde2_e6ec_4b49_a844_d58db0e9a726.slice/crio-81c3733bf963d967cef4b1948373a84b883d86157faddad20ee67441b8d347d2 WatchSource:0}: Error finding container 81c3733bf963d967cef4b1948373a84b883d86157faddad20ee67441b8d347d2: Status 404 returned error can't find the container with id 81c3733bf963d967cef4b1948373a84b883d86157faddad20ee67441b8d347d2 Jul 18 13:45:02.185361 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:02.185318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" event={"ID":"e2765f9f-2e3a-4810-a3e3-52afab8057ed","Type":"ContainerStarted","Data":"ece382e0f26b9e3a35c417e0d1e7cbceadd84a316c6dd056dc95a23e5fd56e13"} Jul 18 13:45:02.187560 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:02.187489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" event={"ID":"5852dde2-e6ec-4b49-a844-d58db0e9a726","Type":"ContainerStarted","Data":"81c3733bf963d967cef4b1948373a84b883d86157faddad20ee67441b8d347d2"} Jul 18 13:45:04.203498 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.203448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" event={"ID":"0097e143-75f2-4a20-bbcc-09c29df116c3","Type":"ContainerStarted","Data":"faf6e2076ae1abbc445476da1d32030c134a47e21e116aded3860c2af9c0432b"} Jul 18 13:45:04.203926 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.203571 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:04.205205 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.205180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" event={"ID":"5852dde2-e6ec-4b49-a844-d58db0e9a726","Type":"ContainerStarted","Data":"ac054ed2724d77bc17bcc8e5526b233be9f46999a1a5163509fc1bfae80c0cc8"} Jul 18 13:45:04.205293 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.205267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:04.206525 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.206504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" event={"ID":"e2765f9f-2e3a-4810-a3e3-52afab8057ed","Type":"ContainerStarted","Data":"8e90f5a6c8de3b3536a7be2a33f305197aa881053092f14f5fb58a589c822932"} Jul 18 13:45:04.206667 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.206610 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:04.219288 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.219179 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" podStartSLOduration=1.81234283 podStartE2EDuration="4.219164972s" totalImagesPullingTime="2.406822142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:45:04.21875914 +0000 UTC m=+754.768323820" watchObservedRunningTime="2026-07-18 13:45:04.219164972 +0000 UTC m=+754.768729653" Jul 18 13:45:04.240072 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.240009 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" podStartSLOduration=0.832128768 podStartE2EDuration="3.239987639s" totalImagesPullingTime="2.407858871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:45:04.238507483 +0000 UTC m=+754.788072176" watchObservedRunningTime="2026-07-18 13:45:04.239987639 +0000 UTC m=+754.789552316" Jul 18 13:45:04.254042 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:04.253990 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" podStartSLOduration=0.76799737 podStartE2EDuration="3.253973525s" totalImagesPullingTime="2.485976155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:45:04.252543882 +0000 UTC m=+754.802108565" watchObservedRunningTime="2026-07-18 13:45:04.253973525 +0000 UTC m=+754.803538240" Jul 18 13:45:09.243281 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:09.243246 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c4a5666-4fc5-4262-9cc9-56679b6d3c39" containerID="7bb6d447282ffb6c5e72ccf147e4107adc79af42c7495dab60e66d151902ff77" exitCode=0 Jul 18 13:45:09.243732 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:09.243325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" event={"ID":"0c4a5666-4fc5-4262-9cc9-56679b6d3c39","Type":"ContainerDied","Data":"7bb6d447282ffb6c5e72ccf147e4107adc79af42c7495dab60e66d151902ff77"} Jul 18 13:45:09.243833 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:09.243816 2564 scope.go:117] "RemoveContainer" containerID="7bb6d447282ffb6c5e72ccf147e4107adc79af42c7495dab60e66d151902ff77" Jul 18 13:45:09.244983 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:09.244953 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:45:09.245045 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:09.245034 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:45:09.245084 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:09.245070 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 13:45:10.228290 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:10.228253 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-689d469879-6d7n7" Jul 18 13:45:10.255142 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:10.255098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" event={"ID":"0c4a5666-4fc5-4262-9cc9-56679b6d3c39","Type":"ContainerStarted","Data":"a377e48d057f88003019f8301f79657c3047d494aa92622ffe623f808447b479"} Jul 18 13:45:10.730522 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:10.730481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:45:11.213479 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:11.213439 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:45:11.214719 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:11.214690 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bk579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 13:45:11.214840 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:11.214759 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" podUID="0c4a5666-4fc5-4262-9cc9-56679b6d3c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 13:45:12.214691 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:12.214654 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bk579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 13:45:12.215087 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:12.214726 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" podUID="0c4a5666-4fc5-4262-9cc9-56679b6d3c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 13:45:13.213706 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:13.213660 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bk579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 13:45:13.213885 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:13.213727 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" podUID="0c4a5666-4fc5-4262-9cc9-56679b6d3c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 13:45:14.214497 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:14.214458 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bk579 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 13:45:14.214859 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:14.214529 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" podUID="0c4a5666-4fc5-4262-9cc9-56679b6d3c39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 13:45:15.217849 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:15.217817 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:45:15.218886 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:15.218859 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bk579" Jul 18 13:45:15.220440 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:15.220418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-frbgv" Jul 18 13:45:15.220576 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:15.220476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-kpt67" Jul 18 13:45:20.295161 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:20.295121 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:45:33.590632 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.590593 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7"] Jul 18 13:45:33.594336 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.594312 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.596945 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.596918 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 18 13:45:33.598024 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.597999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 18 13:45:33.598149 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.598028 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6l8xf\"" Jul 18 13:45:33.598221 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.598096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 18 13:45:33.606704 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.606676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7"] Jul 18 13:45:33.761279 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.761243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.761505 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.761299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhct9\" (UniqueName: \"kubernetes.io/projected/2ace6b59-bd73-4c9c-96a7-175de649ad36-kube-api-access-qhct9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.761505 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.761341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ace6b59-bd73-4c9c-96a7-175de649ad36-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.761505 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.761403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.761505 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.761447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.761505 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.761472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862015 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.861925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862015 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.861974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862216 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.862085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862216 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.862189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862314 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.862233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhct9\" (UniqueName: \"kubernetes.io/projected/2ace6b59-bd73-4c9c-96a7-175de649ad36-kube-api-access-qhct9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862314 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.862265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ace6b59-bd73-4c9c-96a7-175de649ad36-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.862825 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.862787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.863046 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.863024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.864810 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.864782 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.870575 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.868741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ace6b59-bd73-4c9c-96a7-175de649ad36-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.870575 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.869610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ace6b59-bd73-4c9c-96a7-175de649ad36-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.872368 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.872345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhct9\" (UniqueName: \"kubernetes.io/projected/2ace6b59-bd73-4c9c-96a7-175de649ad36-kube-api-access-qhct9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dk7z7\" (UID: \"2ace6b59-bd73-4c9c-96a7-175de649ad36\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:33.905479 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:33.905442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:34.048569 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:34.048538 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7"] Jul 18 13:45:34.050046 ip-10-0-136-156 kubenswrapper[2564]: W0718 13:45:34.050018 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ace6b59_bd73_4c9c_96a7_175de649ad36.slice/crio-686d689b5a912f5a39e2f09af1a1a6eded684618d8072fee1b0a94b336415837 WatchSource:0}: Error finding container 686d689b5a912f5a39e2f09af1a1a6eded684618d8072fee1b0a94b336415837: Status 404 returned error can't find the container with id 686d689b5a912f5a39e2f09af1a1a6eded684618d8072fee1b0a94b336415837 Jul 18 13:45:34.386550 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:34.386512 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:45:34.390238 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:34.390194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" event={"ID":"2ace6b59-bd73-4c9c-96a7-175de649ad36","Type":"ContainerStarted","Data":"686d689b5a912f5a39e2f09af1a1a6eded684618d8072fee1b0a94b336415837"} Jul 18 13:45:35.397297 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:35.397205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" event={"ID":"2ace6b59-bd73-4c9c-96a7-175de649ad36","Type":"ContainerStarted","Data":"6b38507f01d1d5d38405cd38761073d10b962ce1ddd24835f92c1b70beb88021"} Jul 18 13:45:35.416206 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:35.416143 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" podStartSLOduration=1.408046361 podStartE2EDuration="2.416126714s" totalImagesPullingTime="1.008080353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 13:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 13:45:35.413921591 +0000 UTC m=+785.963486276" watchObservedRunningTime="2026-07-18 13:45:35.416126714 +0000 UTC m=+785.965691395" Jul 18 13:45:43.906441 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:43.906407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:43.906960 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:43.906476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:43.919304 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:43.919277 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:44.456918 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:44.456888 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dk7z7" Jul 18 13:45:44.683770 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:44.683732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:45:51.588759 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:51.588676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:45:57.587625 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:45:57.587581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:46:25.387769 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:46:25.387731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:47:52.789608 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:47:52.789560 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:48:04.485396 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:04.485334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:48:11.047676 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.047644 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f744f9d59-2np42"] Jul 18 13:48:11.048111 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.047896 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f744f9d59-2np42" podUID="ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" containerName="manager" containerID="cri-o://7515d7652223e50c7e5b8913e1f7550bb242a390300032e4abdaf9017a3184e3" gracePeriod=30 Jul 18 13:48:11.242091 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.242056 2564 generic.go:356] "Generic (PLEG): container finished" podID="ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" containerID="7515d7652223e50c7e5b8913e1f7550bb242a390300032e4abdaf9017a3184e3" exitCode=0 Jul 18 13:48:11.242246 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.242124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f744f9d59-2np42" event={"ID":"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9","Type":"ContainerDied","Data":"7515d7652223e50c7e5b8913e1f7550bb242a390300032e4abdaf9017a3184e3"} Jul 18 13:48:11.303126 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.303067 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:48:11.414837 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.414785 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz9xc\" (UniqueName: \"kubernetes.io/projected/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-kube-api-access-bz9xc\") pod \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " Jul 18 13:48:11.414837 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.414842 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert\") pod \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\" (UID: \"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9\") " Jul 18 13:48:11.417016 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.416981 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" (UID: "ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 13:48:11.417195 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.417127 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-kube-api-access-bz9xc" (OuterVolumeSpecName: "kube-api-access-bz9xc") pod "ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" (UID: "ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9"). InnerVolumeSpecName "kube-api-access-bz9xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 13:48:11.515527 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.515486 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz9xc\" (UniqueName: \"kubernetes.io/projected/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-kube-api-access-bz9xc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:48:11.515527 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:11.515518 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9-webhook-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 13:48:12.248928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:12.248849 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f744f9d59-2np42" Jul 18 13:48:12.248928 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:12.248886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f744f9d59-2np42" event={"ID":"ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9","Type":"ContainerDied","Data":"4744571df6a672fb231a387f792f72ef18aa4500375e780f99ead540c2c174b1"} Jul 18 13:48:12.249739 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:12.248955 2564 scope.go:117] "RemoveContainer" containerID="7515d7652223e50c7e5b8913e1f7550bb242a390300032e4abdaf9017a3184e3" Jul 18 13:48:12.266421 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:12.266396 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f744f9d59-2np42"] Jul 18 13:48:12.269584 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:12.269559 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f744f9d59-2np42"] Jul 18 13:48:13.292179 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:13.292148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:48:13.972321 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:48:13.972277 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" path="/var/lib/kubelet/pods/ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9/volumes" Jul 18 13:49:53.592449 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:49:53.592409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:50:34.090732 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:50:34.090689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:50:43.184090 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:50:43.184051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:50:51.990532 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:50:51.990495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:51:03.184742 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:51:03.184703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:51:13.191451 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:51:13.191415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:51:24.983022 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:51:24.982986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:51:35.686276 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:51:35.686234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:51:45.394507 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:51:45.394471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:52:10.484891 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:52:10.484806 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:52:20.888169 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:52:20.888135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:52:42.183875 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:52:42.183835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:52:52.591361 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:52:52.591322 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:53:10.091422 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:53:10.091384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:53:19.987334 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:53:19.987296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:53:29.885564 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:53:29.885482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:53:39.587190 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:53:39.587152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:53:49.191980 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:53:49.191940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:55:38.183717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:55:38.183677 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:55:49.679658 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:55:49.679620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:55:58.787480 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:55:58.787442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:56:10.779633 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:56:10.779592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:56:20.578251 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:56:20.578214 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:56:32.185259 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:56:32.185165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:56:40.885717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:56:40.885683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:56:51.978704 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:56:51.978674 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:57:02.684861 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:57:02.684817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:57:11.581346 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:57:11.581313 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:57:22.583717 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:57:22.583684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:57:44.085810 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:57:44.085776 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:57:54.082563 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:57:54.082481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:58:07.983401 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:58:07.983350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:58:21.483690 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:58:21.483657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:58:42.878536 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:58:42.878497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:58:56.587002 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:58:56.586963 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:59:06.499596 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:59:06.499555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:59:18.487915 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:59:18.487879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:59:26.783158 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:59:26.783062 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:59:36.380001 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:59:36.379961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:59:46.494629 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:59:46.494593 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 13:59:55.695261 ip-10-0-136-156 kubenswrapper[2564]: I0718 13:59:55.695211 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:00:00.142151 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.142110 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz"] Jul 18 14:00:00.142588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.142515 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" containerName="manager" Jul 18 14:00:00.142588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.142527 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" containerName="manager" Jul 18 14:00:00.142679 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.142595 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf3a2dc-6a3c-45c4-91cf-ece14ef0d6f9" containerName="manager" Jul 18 14:00:00.145842 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.145821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" Jul 18 14:00:00.148734 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.148713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:00:00.152246 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.152220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz"] Jul 18 14:00:00.158114 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.158097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" Jul 18 14:00:00.299901 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.299867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz"] Jul 18 14:00:00.301869 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:00:00.301841 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53532842_c490_4dad_9c2e_eafb3eddfc73.slice/crio-7d17873cc116e7fdd3617e0fe1196250b8267826c6211cbb9886dd8cebc2c7a5 WatchSource:0}: Error finding container 7d17873cc116e7fdd3617e0fe1196250b8267826c6211cbb9886dd8cebc2c7a5: Status 404 returned error can't find the container with id 7d17873cc116e7fdd3617e0fe1196250b8267826c6211cbb9886dd8cebc2c7a5 Jul 18 14:00:00.304153 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:00.304133 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 14:00:01.135049 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:01.135004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" event={"ID":"53532842-c490-4dad-9c2e-eafb3eddfc73","Type":"ContainerStarted","Data":"7d17873cc116e7fdd3617e0fe1196250b8267826c6211cbb9886dd8cebc2c7a5"} Jul 18 14:00:05.160026 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:05.159931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" event={"ID":"53532842-c490-4dad-9c2e-eafb3eddfc73","Type":"ContainerStarted","Data":"f7b64a5385078535ed38fbb1e84094d9e091b00626d06f86a03a17acb4ed00d3"} Jul 18 14:00:05.176501 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:05.176449 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" podStartSLOduration=0.600062558 podStartE2EDuration="5.17643182s" totalImagesPullingTime="4.576369262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:00:05.174193313 +0000 UTC m=+1655.723758003" watchObservedRunningTime="2026-07-18 14:00:05.17643182 +0000 UTC m=+1655.725996501" Jul 18 14:00:06.166613 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:06.166516 2564 generic.go:356] "Generic (PLEG): container finished" podID="53532842-c490-4dad-9c2e-eafb3eddfc73" containerID="f7b64a5385078535ed38fbb1e84094d9e091b00626d06f86a03a17acb4ed00d3" exitCode=0 Jul 18 14:00:06.166992 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:06.166605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" event={"ID":"53532842-c490-4dad-9c2e-eafb3eddfc73","Type":"ContainerDied","Data":"f7b64a5385078535ed38fbb1e84094d9e091b00626d06f86a03a17acb4ed00d3"} Jul 18 14:00:06.979351 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:06.979314 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:00:07.306490 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:07.306467 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" Jul 18 14:00:08.179314 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:08.179286 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" Jul 18 14:00:08.179504 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:08.179327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739720-xbzvz" event={"ID":"53532842-c490-4dad-9c2e-eafb3eddfc73","Type":"ContainerDied","Data":"7d17873cc116e7fdd3617e0fe1196250b8267826c6211cbb9886dd8cebc2c7a5"} Jul 18 14:00:08.179504 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:08.179396 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d17873cc116e7fdd3617e0fe1196250b8267826c6211cbb9886dd8cebc2c7a5" Jul 18 14:00:18.483715 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:18.483675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:00:32.985736 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:32.985690 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:00:42.689030 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:42.688991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:00:49.282916 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:49.282880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:00:59.992891 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:00:59.992794 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:01:07.889014 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:01:07.888973 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:01:19.988987 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:01:19.988940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:01:31.890956 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:01:31.890917 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:01:41.285885 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:01:41.285848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:01:46.984988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:01:46.984953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:02:00.907527 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.907489 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7"] Jul 18 14:02:00.907961 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.907937 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53532842-c490-4dad-9c2e-eafb3eddfc73" containerName="cleanup" Jul 18 14:02:00.907961 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.907952 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53532842-c490-4dad-9c2e-eafb3eddfc73" containerName="cleanup" Jul 18 14:02:00.908042 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.908033 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53532842-c490-4dad-9c2e-eafb3eddfc73" containerName="cleanup" Jul 18 14:02:00.911343 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.911324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:00.914519 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.914494 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7d23fa97-serving-cert\"" Jul 18 14:02:00.924548 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:00.924517 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7"] Jul 18 14:02:01.021435 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.021369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtqf\" (UniqueName: \"kubernetes.io/projected/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-kube-api-access-hxtqf\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.021615 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.021446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.021615 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.021477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-openshift-service-ca\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.123070 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.123025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtqf\" (UniqueName: \"kubernetes.io/projected/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-kube-api-access-hxtqf\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.123070 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.123073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.123334 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.123103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-openshift-service-ca\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.123334 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:02:01.123212 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-serving-cert: secret "maas-api-e2e-ait-7d23fa97-serving-cert" not found Jul 18 14:02:01.123334 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:02:01.123287 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls podName:311494a0-b9e0-47a2-ab8d-544c1a4b04a7 nodeName:}" failed. No retries permitted until 2026-07-18 14:02:01.62326839 +0000 UTC m=+1772.172833053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls") pod "maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" (UID: "311494a0-b9e0-47a2-ab8d-544c1a4b04a7") : secret "maas-api-e2e-ait-7d23fa97-serving-cert" not found Jul 18 14:02:01.123876 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.123853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-openshift-service-ca\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.135122 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.135095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtqf\" (UniqueName: \"kubernetes.io/projected/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-kube-api-access-hxtqf\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.628071 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.628025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.630512 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.630493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls\") pod \"maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:01.823241 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:01.823208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:02.172033 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:02.172005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7"] Jul 18 14:02:02.173435 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:02:02.173356 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311494a0_b9e0_47a2_ab8d_544c1a4b04a7.slice/crio-df9fdd85e33be4d6acdbab8d7ac1cc1a36b3f84bbd0d7ec644ca4b373e0bce7c WatchSource:0}: Error finding container df9fdd85e33be4d6acdbab8d7ac1cc1a36b3f84bbd0d7ec644ca4b373e0bce7c: Status 404 returned error can't find the container with id df9fdd85e33be4d6acdbab8d7ac1cc1a36b3f84bbd0d7ec644ca4b373e0bce7c Jul 18 14:02:02.810586 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:02.810556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" event={"ID":"311494a0-b9e0-47a2-ab8d-544c1a4b04a7","Type":"ContainerStarted","Data":"df9fdd85e33be4d6acdbab8d7ac1cc1a36b3f84bbd0d7ec644ca4b373e0bce7c"} Jul 18 14:02:03.817558 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:03.817519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" event={"ID":"311494a0-b9e0-47a2-ab8d-544c1a4b04a7","Type":"ContainerStarted","Data":"bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d"} Jul 18 14:02:03.817954 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:03.817606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:03.835452 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:03.835396 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" podStartSLOduration=3.201746558 podStartE2EDuration="3.8353552s" totalImagesPullingTime="633.608642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:02:03.83345826 +0000 UTC m=+1774.383022943" watchObservedRunningTime="2026-07-18 14:02:03.8353552 +0000 UTC m=+1774.384919895" Jul 18 14:02:07.586614 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:07.586571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:02:09.828833 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:09.828803 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:36.667981 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.667945 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7"] Jul 18 14:02:36.668543 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.668185 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" podUID="311494a0-b9e0-47a2-ab8d-544c1a4b04a7" containerName="maas-api" containerID="cri-o://bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d" gracePeriod=30 Jul 18 14:02:36.916027 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.915996 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:36.957626 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.957593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxtqf\" (UniqueName: \"kubernetes.io/projected/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-kube-api-access-hxtqf\") pod \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " Jul 18 14:02:36.957824 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.957694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls\") pod \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " Jul 18 14:02:36.957824 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.957723 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-openshift-service-ca\") pod \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\" (UID: \"311494a0-b9e0-47a2-ab8d-544c1a4b04a7\") " Jul 18 14:02:36.958150 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.958125 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "311494a0-b9e0-47a2-ab8d-544c1a4b04a7" (UID: "311494a0-b9e0-47a2-ab8d-544c1a4b04a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:02:36.959885 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.959856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-kube-api-access-hxtqf" (OuterVolumeSpecName: "kube-api-access-hxtqf") pod "311494a0-b9e0-47a2-ab8d-544c1a4b04a7" (UID: "311494a0-b9e0-47a2-ab8d-544c1a4b04a7"). InnerVolumeSpecName "kube-api-access-hxtqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:02:36.959885 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.959862 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "311494a0-b9e0-47a2-ab8d-544c1a4b04a7" (UID: "311494a0-b9e0-47a2-ab8d-544c1a4b04a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 14:02:36.995712 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.995676 2564 generic.go:356] "Generic (PLEG): container finished" podID="311494a0-b9e0-47a2-ab8d-544c1a4b04a7" containerID="bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d" exitCode=0 Jul 18 14:02:36.995858 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.995727 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" Jul 18 14:02:36.995858 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.995761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" event={"ID":"311494a0-b9e0-47a2-ab8d-544c1a4b04a7","Type":"ContainerDied","Data":"bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d"} Jul 18 14:02:36.995858 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.995818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7" event={"ID":"311494a0-b9e0-47a2-ab8d-544c1a4b04a7","Type":"ContainerDied","Data":"df9fdd85e33be4d6acdbab8d7ac1cc1a36b3f84bbd0d7ec644ca4b373e0bce7c"} Jul 18 14:02:36.995858 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:36.995842 2564 scope.go:117] "RemoveContainer" containerID="bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d" Jul 18 14:02:37.005511 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.005472 2564 scope.go:117] "RemoveContainer" containerID="bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d" Jul 18 14:02:37.005746 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:02:37.005727 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d\": container with ID starting with bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d not found: ID does not exist" containerID="bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d" Jul 18 14:02:37.005789 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.005756 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d"} err="failed to get container status \"bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d\": rpc error: code = NotFound desc = could not find container \"bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d\": container with ID starting with bb5dfd50411b3ed36ad977a9f912f965fa5d33eb223d656de38ae45daf692c9d not found: ID does not exist" Jul 18 14:02:37.017272 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.017249 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7"] Jul 18 14:02:37.020989 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.020963 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7d23fa97-78dcb89c6f-mlcq7"] Jul 18 14:02:37.059578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.059322 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxtqf\" (UniqueName: \"kubernetes.io/projected/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-kube-api-access-hxtqf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:02:37.059578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.059352 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:02:37.059578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.059366 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311494a0-b9e0-47a2-ab8d-544c1a4b04a7-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:02:37.973367 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:37.973332 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311494a0-b9e0-47a2-ab8d-544c1a4b04a7" path="/var/lib/kubelet/pods/311494a0-b9e0-47a2-ab8d-544c1a4b04a7/volumes" Jul 18 14:02:39.490534 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:39.490498 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:02:42.551217 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.551172 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl"] Jul 18 14:02:42.551783 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.551762 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311494a0-b9e0-47a2-ab8d-544c1a4b04a7" containerName="maas-api" Jul 18 14:02:42.551865 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.551787 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="311494a0-b9e0-47a2-ab8d-544c1a4b04a7" containerName="maas-api" Jul 18 14:02:42.554868 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.552488 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="311494a0-b9e0-47a2-ab8d-544c1a4b04a7" containerName="maas-api" Jul 18 14:02:42.561114 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.561085 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.563742 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.563714 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ecedf3e3-gw-openshift-default-dockercfg-s4fgg\"" Jul 18 14:02:42.568484 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.568457 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl"] Jul 18 14:02:42.611306 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-certs\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611306 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-podinfo\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-token\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-socket\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18112e10-53ab-4093-a7c4-ef18b58d6452-istiod-ca-cert\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-data\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-credential-socket\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhtq\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-kube-api-access-kfhtq\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.611832 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.611592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-envoy\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712153 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-socket\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712153 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18112e10-53ab-4093-a7c4-ef18b58d6452-istiod-ca-cert\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-data\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-credential-socket\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhtq\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-kube-api-access-kfhtq\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-envoy\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-certs\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-podinfo\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712735 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-token\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712735 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-socket\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712847 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712782 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-credential-socket\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712907 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-certs\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.712956 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-data\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.713005 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.712989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18112e10-53ab-4093-a7c4-ef18b58d6452-istiod-ca-cert\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.714740 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.714710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-envoy\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.714975 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.714955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-podinfo\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.720135 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.720109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-token\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.720978 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.720955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhtq\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-kube-api-access-kfhtq\") pod \"e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:42.875176 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:42.875082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:43.018410 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:43.018363 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl"] Jul 18 14:02:43.019530 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:02:43.019505 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18112e10_53ab_4093_a7c4_ef18b58d6452.slice/crio-0cb515885e1def1f75df0ae61485a00e3b4e2252dd40124462733c2c84343a00 WatchSource:0}: Error finding container 0cb515885e1def1f75df0ae61485a00e3b4e2252dd40124462733c2c84343a00: Status 404 returned error can't find the container with id 0cb515885e1def1f75df0ae61485a00e3b4e2252dd40124462733c2c84343a00 Jul 18 14:02:43.021785 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:43.021753 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:02:43.021849 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:43.021821 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:02:43.021887 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:43.021848 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:02:43.030014 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:43.029986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" event={"ID":"18112e10-53ab-4093-a7c4-ef18b58d6452","Type":"ContainerStarted","Data":"0cb515885e1def1f75df0ae61485a00e3b4e2252dd40124462733c2c84343a00"} Jul 18 14:02:44.036423 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:44.036382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" event={"ID":"18112e10-53ab-4093-a7c4-ef18b58d6452","Type":"ContainerStarted","Data":"2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6"} Jul 18 14:02:44.055417 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:44.055346 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podStartSLOduration=2.055329864 podStartE2EDuration="2.055329864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:02:44.054839888 +0000 UTC m=+1814.604404569" watchObservedRunningTime="2026-07-18 14:02:44.055329864 +0000 UTC m=+1814.604894545" Jul 18 14:02:44.875847 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:44.875807 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:44.881070 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:44.881043 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:45.042389 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.042351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:45.043416 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.043395 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:02:45.592433 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.592398 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6"] Jul 18 14:02:45.596043 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.596023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.598354 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.598333 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ecedf3e3-serving-cert\"" Jul 18 14:02:45.605043 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.604527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6"] Jul 18 14:02:45.638881 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.638840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zll2q\" (UniqueName: \"kubernetes.io/projected/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-kube-api-access-zll2q\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.639034 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.638942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.639034 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.639002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-openshift-service-ca\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.739854 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.739817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-openshift-service-ca\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.740084 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.739875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zll2q\" (UniqueName: \"kubernetes.io/projected/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-kube-api-access-zll2q\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.740084 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.739942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.740084 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:02:45.740038 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-serving-cert: secret "maas-api-e2e-migrate-ecedf3e3-serving-cert" not found Jul 18 14:02:45.740196 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:02:45.740104 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls podName:7f90a543-9ba3-48a5-9c5b-8ecb4e52df97 nodeName:}" failed. No retries permitted until 2026-07-18 14:02:46.2400866 +0000 UTC m=+1816.789651259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls") pod "maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" (UID: "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97") : secret "maas-api-e2e-migrate-ecedf3e3-serving-cert" not found Jul 18 14:02:45.740556 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.740538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-openshift-service-ca\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:45.762617 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:45.762589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zll2q\" (UniqueName: \"kubernetes.io/projected/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-kube-api-access-zll2q\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:46.246135 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:46.246095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:46.248509 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:46.248487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls\") pod \"maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:46.509765 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:46.509657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:46.650485 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:46.650455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6"] Jul 18 14:02:46.651705 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:02:46.651670 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f90a543_9ba3_48a5_9c5b_8ecb4e52df97.slice/crio-2bcab316c89a2fda2c8d8efa5e81ebafac57be6a7efaa1deda8906c0169c8367 WatchSource:0}: Error finding container 2bcab316c89a2fda2c8d8efa5e81ebafac57be6a7efaa1deda8906c0169c8367: Status 404 returned error can't find the container with id 2bcab316c89a2fda2c8d8efa5e81ebafac57be6a7efaa1deda8906c0169c8367 Jul 18 14:02:47.054054 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:47.054022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" event={"ID":"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97","Type":"ContainerStarted","Data":"2bcab316c89a2fda2c8d8efa5e81ebafac57be6a7efaa1deda8906c0169c8367"} Jul 18 14:02:48.060549 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:48.060505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" event={"ID":"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97","Type":"ContainerStarted","Data":"0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f"} Jul 18 14:02:48.060966 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:48.060673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:02:48.078435 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:48.078354 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" podStartSLOduration=2.704326515 podStartE2EDuration="3.078340325s" totalImagesPullingTime="374.01381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:02:48.076083405 +0000 UTC m=+1818.625648085" watchObservedRunningTime="2026-07-18 14:02:48.078340325 +0000 UTC m=+1818.627905005" Jul 18 14:02:51.689653 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:51.689616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:02:54.073233 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:02:54.073190 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:03:00.044276 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.044233 2564 patch_prober.go:28] interesting pod/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 18 14:03:00.044727 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.044290 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 18 14:03:00.135039 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.135004 2564 generic.go:356] "Generic (PLEG): container finished" podID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerID="2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6" exitCode=0 Jul 18 14:03:00.135208 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.135076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" event={"ID":"18112e10-53ab-4093-a7c4-ef18b58d6452","Type":"ContainerDied","Data":"2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6"} Jul 18 14:03:00.135510 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.135495 2564 scope.go:117] "RemoveContainer" containerID="2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6" Jul 18 14:03:00.136628 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.136597 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:03:00.136706 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.136686 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:03:00.136748 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:00.136734 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:03:01.148506 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:01.148466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" event={"ID":"18112e10-53ab-4093-a7c4-ef18b58d6452","Type":"ContainerStarted","Data":"9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d"} Jul 18 14:03:01.876185 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:01.876142 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:03:01.876368 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:01.876348 2564 patch_prober.go:28] interesting pod/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 18 14:03:01.876444 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:01.876415 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 18 14:03:02.875702 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:02.875664 2564 patch_prober.go:28] interesting pod/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 18 14:03:02.876098 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:02.875719 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 18 14:03:03.875923 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:03.875872 2564 patch_prober.go:28] interesting pod/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 18 14:03:03.876335 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:03.875954 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 18 14:03:04.876531 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:04.876487 2564 patch_prober.go:28] interesting pod/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 18 14:03:04.876922 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:04.876554 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 18 14:03:05.879611 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:05.879578 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:03:05.879993 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:05.879839 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:03:05.880670 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:05.880652 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:03:20.843984 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:20.843943 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6"] Jul 18 14:03:20.844551 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:20.844207 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" podUID="7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" containerName="maas-api" containerID="cri-o://0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f" gracePeriod=30 Jul 18 14:03:21.093272 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.093247 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:03:21.183014 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.182930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zll2q\" (UniqueName: \"kubernetes.io/projected/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-kube-api-access-zll2q\") pod \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " Jul 18 14:03:21.183014 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.182983 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls\") pod \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " Jul 18 14:03:21.183231 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.183020 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-openshift-service-ca\") pod \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\" (UID: \"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97\") " Jul 18 14:03:21.183534 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.183501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" (UID: "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:03:21.185405 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.185351 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" (UID: "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 14:03:21.185535 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.185353 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-kube-api-access-zll2q" (OuterVolumeSpecName: "kube-api-access-zll2q") pod "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" (UID: "7f90a543-9ba3-48a5-9c5b-8ecb4e52df97"). InnerVolumeSpecName "kube-api-access-zll2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:03:21.265728 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.265689 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" containerID="0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f" exitCode=0 Jul 18 14:03:21.265898 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.265739 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" Jul 18 14:03:21.265898 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.265781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" event={"ID":"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97","Type":"ContainerDied","Data":"0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f"} Jul 18 14:03:21.265898 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.265848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6" event={"ID":"7f90a543-9ba3-48a5-9c5b-8ecb4e52df97","Type":"ContainerDied","Data":"2bcab316c89a2fda2c8d8efa5e81ebafac57be6a7efaa1deda8906c0169c8367"} Jul 18 14:03:21.265898 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.265873 2564 scope.go:117] "RemoveContainer" containerID="0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f" Jul 18 14:03:21.275570 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.275549 2564 scope.go:117] "RemoveContainer" containerID="0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f" Jul 18 14:03:21.275819 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:03:21.275798 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f\": container with ID starting with 0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f not found: ID does not exist" containerID="0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f" Jul 18 14:03:21.275898 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.275832 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f"} err="failed to get container status \"0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f\": rpc error: code = NotFound desc = could not find container \"0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f\": container with ID starting with 0c0d4257c59c0a2f1b624b3ec78de24a951045dfea28df776170fdda5d08b59f not found: ID does not exist" Jul 18 14:03:21.284581 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.284511 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:21.284581 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.284542 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:21.284581 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.284557 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zll2q\" (UniqueName: \"kubernetes.io/projected/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97-kube-api-access-zll2q\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:21.287068 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.287044 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6"] Jul 18 14:03:21.289464 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.289439 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecedf3e3-5d5df77845-ln9z6"] Jul 18 14:03:21.972930 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:21.972894 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" path="/var/lib/kubelet/pods/7f90a543-9ba3-48a5-9c5b-8ecb4e52df97/volumes" Jul 18 14:03:23.594591 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:23.594554 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:03:25.705391 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:25.705344 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl"] Jul 18 14:03:25.705773 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:25.705671 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" containerID="cri-o://9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d" gracePeriod=30 Jul 18 14:03:29.358810 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.358737 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb"] Jul 18 14:03:29.359337 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.359318 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" containerName="maas-api" Jul 18 14:03:29.359418 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.359339 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" containerName="maas-api" Jul 18 14:03:29.359499 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.359438 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f90a543-9ba3-48a5-9c5b-8ecb4e52df97" containerName="maas-api" Jul 18 14:03:29.363947 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.363930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.366509 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.366482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-56e18679-serving-cert\"" Jul 18 14:03:29.373854 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.373825 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb"] Jul 18 14:03:29.460775 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.460731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe2a42a-8579-4468-825a-16984e1904d1-maas-api-tls\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.460982 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.460793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a42a-8579-4468-825a-16984e1904d1-openshift-service-ca\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.460982 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.460843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pwt\" (UniqueName: \"kubernetes.io/projected/7fe2a42a-8579-4468-825a-16984e1904d1-kube-api-access-82pwt\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.562464 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.562428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pwt\" (UniqueName: \"kubernetes.io/projected/7fe2a42a-8579-4468-825a-16984e1904d1-kube-api-access-82pwt\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.562668 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.562519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe2a42a-8579-4468-825a-16984e1904d1-maas-api-tls\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.562668 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.562549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a42a-8579-4468-825a-16984e1904d1-openshift-service-ca\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.563264 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.563244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a42a-8579-4468-825a-16984e1904d1-openshift-service-ca\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.564992 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.564973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe2a42a-8579-4468-825a-16984e1904d1-maas-api-tls\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.570704 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.570684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pwt\" (UniqueName: \"kubernetes.io/projected/7fe2a42a-8579-4468-825a-16984e1904d1-kube-api-access-82pwt\") pod \"maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.675369 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.675292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:29.815438 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:29.815412 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb"] Jul 18 14:03:29.817652 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:03:29.817610 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe2a42a_8579_4468_825a_16984e1904d1.slice/crio-1f98e44e98a5908d68bb55ec3e4124726cb4066775b94181623d7bfd2c5084f6 WatchSource:0}: Error finding container 1f98e44e98a5908d68bb55ec3e4124726cb4066775b94181623d7bfd2c5084f6: Status 404 returned error can't find the container with id 1f98e44e98a5908d68bb55ec3e4124726cb4066775b94181623d7bfd2c5084f6 Jul 18 14:03:30.317242 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:30.317203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" event={"ID":"7fe2a42a-8579-4468-825a-16984e1904d1","Type":"ContainerStarted","Data":"1f98e44e98a5908d68bb55ec3e4124726cb4066775b94181623d7bfd2c5084f6"} Jul 18 14:03:30.957604 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:30.957576 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:03:31.078813 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-data\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.078813 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078814 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-podinfo\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-token\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078849 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-envoy\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-socket\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-certs\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-credential-socket\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18112e10-53ab-4093-a7c4-ef18b58d6452-istiod-ca-cert\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079048 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.078999 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfhtq\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-kube-api-access-kfhtq\") pod \"18112e10-53ab-4093-a7c4-ef18b58d6452\" (UID: \"18112e10-53ab-4093-a7c4-ef18b58d6452\") " Jul 18 14:03:31.079464 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079190 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:03:31.079464 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079203 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:03:31.079464 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079328 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:03:31.079686 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079524 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18112e10-53ab-4093-a7c4-ef18b58d6452-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:03:31.079686 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079565 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.079686 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079587 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.079686 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.079668 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-data" (OuterVolumeSpecName: "istio-data") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:03:31.081193 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.081162 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-token" (OuterVolumeSpecName: "istio-token") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:03:31.081316 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.081252 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 14:03:31.081316 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.081262 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-kube-api-access-kfhtq" (OuterVolumeSpecName: "kube-api-access-kfhtq") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "kube-api-access-kfhtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:03:31.081615 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.081597 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18112e10-53ab-4093-a7c4-ef18b58d6452" (UID: "18112e10-53ab-4093-a7c4-ef18b58d6452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:03:31.180321 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180284 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.180321 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180313 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18112e10-53ab-4093-a7c4-ef18b58d6452-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.180321 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180323 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhtq\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-kube-api-access-kfhtq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.180321 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180332 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.180588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180340 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.180588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180347 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.180588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.180355 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18112e10-53ab-4093-a7c4-ef18b58d6452-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:03:31.325232 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.325197 2564 generic.go:356] "Generic (PLEG): container finished" podID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerID="9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d" exitCode=0 Jul 18 14:03:31.325457 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.325245 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" Jul 18 14:03:31.325457 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.325281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" event={"ID":"18112e10-53ab-4093-a7c4-ef18b58d6452","Type":"ContainerDied","Data":"9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d"} Jul 18 14:03:31.325457 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.325347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl" event={"ID":"18112e10-53ab-4093-a7c4-ef18b58d6452","Type":"ContainerDied","Data":"0cb515885e1def1f75df0ae61485a00e3b4e2252dd40124462733c2c84343a00"} Jul 18 14:03:31.325457 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.325401 2564 scope.go:117] "RemoveContainer" containerID="9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d" Jul 18 14:03:31.326920 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.326897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" event={"ID":"7fe2a42a-8579-4468-825a-16984e1904d1","Type":"ContainerStarted","Data":"1a7e33b9b68f431b1c0fc79094a5aed7ee916c0101a6dfbee17345f21819f1ba"} Jul 18 14:03:31.327038 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.327020 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:03:31.335666 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.335649 2564 scope.go:117] "RemoveContainer" containerID="2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6" Jul 18 14:03:31.343954 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.343933 2564 scope.go:117] "RemoveContainer" containerID="9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d" Jul 18 14:03:31.344232 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:03:31.344204 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d\": container with ID starting with 9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d not found: ID does not exist" containerID="9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d" Jul 18 14:03:31.344340 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.344240 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d"} err="failed to get container status \"9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d\": rpc error: code = NotFound desc = could not find container \"9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d\": container with ID starting with 9681160cb8df25f4e16a8f20a35be2b393d315c2ac5b3a44a879ad33fd08c79d not found: ID does not exist" Jul 18 14:03:31.344340 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.344265 2564 scope.go:117] "RemoveContainer" containerID="2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6" Jul 18 14:03:31.344616 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:03:31.344598 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6\": container with ID starting with 2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6 not found: ID does not exist" containerID="2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6" Jul 18 14:03:31.344704 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.344621 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6"} err="failed to get container status \"2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6\": rpc error: code = NotFound desc = could not find container \"2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6\": container with ID starting with 2e8d18fc2930f9ce425194a20938750ce6599837880f0fca62c31b43275c76b6 not found: ID does not exist" Jul 18 14:03:31.347030 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.346988 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" podStartSLOduration=1.7943434900000002 podStartE2EDuration="2.346975117s" totalImagesPullingTime="552.631627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:03:31.344636105 +0000 UTC m=+1861.894200787" watchObservedRunningTime="2026-07-18 14:03:31.346975117 +0000 UTC m=+1861.896539797" Jul 18 14:03:31.358321 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.358295 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl"] Jul 18 14:03:31.365537 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.365507 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ecedf3e3-gw-openshift-default-6b5cf9877f-bl5nl"] Jul 18 14:03:31.975500 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:31.975467 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" path="/var/lib/kubelet/pods/18112e10-53ab-4093-a7c4-ef18b58d6452/volumes" Jul 18 14:03:36.490092 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:36.490057 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:03:37.339995 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:03:37.339962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:04:05.306609 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.306529 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb"] Jul 18 14:04:05.307163 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.306780 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" podUID="7fe2a42a-8579-4468-825a-16984e1904d1" containerName="maas-api" containerID="cri-o://1a7e33b9b68f431b1c0fc79094a5aed7ee916c0101a6dfbee17345f21819f1ba" gracePeriod=30 Jul 18 14:04:05.533404 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.533349 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fe2a42a-8579-4468-825a-16984e1904d1" containerID="1a7e33b9b68f431b1c0fc79094a5aed7ee916c0101a6dfbee17345f21819f1ba" exitCode=0 Jul 18 14:04:05.533572 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.533408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" event={"ID":"7fe2a42a-8579-4468-825a-16984e1904d1","Type":"ContainerDied","Data":"1a7e33b9b68f431b1c0fc79094a5aed7ee916c0101a6dfbee17345f21819f1ba"} Jul 18 14:04:05.564643 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.564582 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:04:05.603558 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.603518 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a42a-8579-4468-825a-16984e1904d1-openshift-service-ca\") pod \"7fe2a42a-8579-4468-825a-16984e1904d1\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " Jul 18 14:04:05.603755 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.603599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe2a42a-8579-4468-825a-16984e1904d1-maas-api-tls\") pod \"7fe2a42a-8579-4468-825a-16984e1904d1\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " Jul 18 14:04:05.603755 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.603663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82pwt\" (UniqueName: \"kubernetes.io/projected/7fe2a42a-8579-4468-825a-16984e1904d1-kube-api-access-82pwt\") pod \"7fe2a42a-8579-4468-825a-16984e1904d1\" (UID: \"7fe2a42a-8579-4468-825a-16984e1904d1\") " Jul 18 14:04:05.603917 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.603891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe2a42a-8579-4468-825a-16984e1904d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fe2a42a-8579-4468-825a-16984e1904d1" (UID: "7fe2a42a-8579-4468-825a-16984e1904d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:04:05.605843 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.605805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe2a42a-8579-4468-825a-16984e1904d1-kube-api-access-82pwt" (OuterVolumeSpecName: "kube-api-access-82pwt") pod "7fe2a42a-8579-4468-825a-16984e1904d1" (UID: "7fe2a42a-8579-4468-825a-16984e1904d1"). InnerVolumeSpecName "kube-api-access-82pwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:04:05.605843 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.605819 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe2a42a-8579-4468-825a-16984e1904d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fe2a42a-8579-4468-825a-16984e1904d1" (UID: "7fe2a42a-8579-4468-825a-16984e1904d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 14:04:05.704901 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.704859 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a42a-8579-4468-825a-16984e1904d1-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:05.704901 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.704889 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe2a42a-8579-4468-825a-16984e1904d1-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:05.704901 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:05.704900 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pwt\" (UniqueName: \"kubernetes.io/projected/7fe2a42a-8579-4468-825a-16984e1904d1-kube-api-access-82pwt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:06.539606 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:06.539573 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" Jul 18 14:04:06.539606 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:06.539596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb" event={"ID":"7fe2a42a-8579-4468-825a-16984e1904d1","Type":"ContainerDied","Data":"1f98e44e98a5908d68bb55ec3e4124726cb4066775b94181623d7bfd2c5084f6"} Jul 18 14:04:06.540062 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:06.539672 2564 scope.go:117] "RemoveContainer" containerID="1a7e33b9b68f431b1c0fc79094a5aed7ee916c0101a6dfbee17345f21819f1ba" Jul 18 14:04:06.557047 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:06.557020 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb"] Jul 18 14:04:06.559201 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:06.559179 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56e18679-dddc5f7c9-kr4kb"] Jul 18 14:04:07.973362 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:07.973324 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe2a42a-8579-4468-825a-16984e1904d1" path="/var/lib/kubelet/pods/7fe2a42a-8579-4468-825a-16984e1904d1/volumes" Jul 18 14:04:08.284110 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:08.284028 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:04:11.859012 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.858975 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm"] Jul 18 14:04:11.859488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859406 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe2a42a-8579-4468-825a-16984e1904d1" containerName="maas-api" Jul 18 14:04:11.859488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859418 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe2a42a-8579-4468-825a-16984e1904d1" containerName="maas-api" Jul 18 14:04:11.859488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859433 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" Jul 18 14:04:11.859488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859439 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" Jul 18 14:04:11.859488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859456 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" Jul 18 14:04:11.859488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859461 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" Jul 18 14:04:11.859703 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859518 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" Jul 18 14:04:11.859703 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.859529 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe2a42a-8579-4468-825a-16984e1904d1" containerName="maas-api" Jul 18 14:04:11.864031 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.864011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.866873 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.866853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ea60214d-openshift-default-dockercfg-w6pw5\"" Jul 18 14:04:11.874610 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.874541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm"] Jul 18 14:04:11.964334 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-certs\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964528 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-envoy\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964528 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-credential-socket\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964528 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-socket\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964649 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-token\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964649 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v477j\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-kube-api-access-v477j\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964649 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94582a77-0c65-4c3a-bca1-f1108cfc7961-istiod-ca-cert\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964759 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-podinfo\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:11.964759 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:11.964751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-data\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-socket\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-token\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066339 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v477j\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-kube-api-access-v477j\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066339 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94582a77-0c65-4c3a-bca1-f1108cfc7961-istiod-ca-cert\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066339 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-podinfo\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066339 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-data\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066339 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-certs\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066632 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-envoy\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066632 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-credential-socket\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066632 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-socket\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066732 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-data\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066732 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-credential-socket\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.066810 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.066749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-certs\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.067170 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.067151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94582a77-0c65-4c3a-bca1-f1108cfc7961-istiod-ca-cert\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.068584 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.068566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-envoy\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.068670 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.068653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-podinfo\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.073673 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.073645 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-token\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.073782 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.073688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v477j\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-kube-api-access-v477j\") pod \"e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.175824 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.175734 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:12.317397 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.317351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm"] Jul 18 14:04:12.318796 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:04:12.318768 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94582a77_0c65_4c3a_bca1_f1108cfc7961.slice/crio-120038eed72a8cdfdbb1d25b43d7ab2af09079d48a49d7f7f013d9e694e8ecad WatchSource:0}: Error finding container 120038eed72a8cdfdbb1d25b43d7ab2af09079d48a49d7f7f013d9e694e8ecad: Status 404 returned error can't find the container with id 120038eed72a8cdfdbb1d25b43d7ab2af09079d48a49d7f7f013d9e694e8ecad Jul 18 14:04:12.321329 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.321294 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:04:12.321446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.321358 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:04:12.321446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.321398 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:04:12.576412 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.576353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" event={"ID":"94582a77-0c65-4c3a-bca1-f1108cfc7961","Type":"ContainerStarted","Data":"35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e"} Jul 18 14:04:12.576588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.576429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" event={"ID":"94582a77-0c65-4c3a-bca1-f1108cfc7961","Type":"ContainerStarted","Data":"120038eed72a8cdfdbb1d25b43d7ab2af09079d48a49d7f7f013d9e694e8ecad"} Jul 18 14:04:12.597301 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:12.597248 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podStartSLOduration=1.597233964 podStartE2EDuration="1.597233964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:04:12.596134663 +0000 UTC m=+1903.145699346" watchObservedRunningTime="2026-07-18 14:04:12.597233964 +0000 UTC m=+1903.146798645" Jul 18 14:04:13.176052 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:13.176017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:13.181026 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:13.181000 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:13.582270 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:13.582238 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:13.583145 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:13.583126 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:13.996968 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:13.996842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z"] Jul 18 14:04:13.997653 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:13.997615 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="18112e10-53ab-4093-a7c4-ef18b58d6452" containerName="istio-proxy" Jul 18 14:04:14.000881 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.000858 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.003729 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.003705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ea60214d-serving-cert\"" Jul 18 14:04:14.013920 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.013890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z"] Jul 18 14:04:14.084486 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.084450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22a7439-efbd-45a0-a7a6-6043fd518238-openshift-service-ca\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.084671 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.084603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9lx\" (UniqueName: \"kubernetes.io/projected/b22a7439-efbd-45a0-a7a6-6043fd518238-kube-api-access-xr9lx\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.084722 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.084669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.185220 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.185183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9lx\" (UniqueName: \"kubernetes.io/projected/b22a7439-efbd-45a0-a7a6-6043fd518238-kube-api-access-xr9lx\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.185722 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.185236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.185722 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.185264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22a7439-efbd-45a0-a7a6-6043fd518238-openshift-service-ca\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.185722 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:04:14.185472 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-serving-cert: secret "maas-api-e2e-derive-ea60214d-serving-cert" not found Jul 18 14:04:14.185722 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:04:14.185566 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls podName:b22a7439-efbd-45a0-a7a6-6043fd518238 nodeName:}" failed. No retries permitted until 2026-07-18 14:04:14.685545147 +0000 UTC m=+1905.235109819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls") pod "maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" (UID: "b22a7439-efbd-45a0-a7a6-6043fd518238") : secret "maas-api-e2e-derive-ea60214d-serving-cert" not found Jul 18 14:04:14.186000 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.185979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22a7439-efbd-45a0-a7a6-6043fd518238-openshift-service-ca\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.199255 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.199222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9lx\" (UniqueName: \"kubernetes.io/projected/b22a7439-efbd-45a0-a7a6-6043fd518238-kube-api-access-xr9lx\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.690635 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.690598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.693144 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.693117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls\") pod \"maas-api-e2e-derive-ea60214d-5cdc78587b-h578z\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:14.916280 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:14.916242 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:15.052166 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:15.052139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z"] Jul 18 14:04:15.054273 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:04:15.054243 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22a7439_efbd_45a0_a7a6_6043fd518238.slice/crio-3537217fea4133c9b74d544b5f3398f03d7c93cc156c007f074f76fc8e06fd1d WatchSource:0}: Error finding container 3537217fea4133c9b74d544b5f3398f03d7c93cc156c007f074f76fc8e06fd1d: Status 404 returned error can't find the container with id 3537217fea4133c9b74d544b5f3398f03d7c93cc156c007f074f76fc8e06fd1d Jul 18 14:04:15.601588 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:15.601544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" event={"ID":"b22a7439-efbd-45a0-a7a6-6043fd518238","Type":"ContainerStarted","Data":"777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa"} Jul 18 14:04:15.602079 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:15.601647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" event={"ID":"b22a7439-efbd-45a0-a7a6-6043fd518238","Type":"ContainerStarted","Data":"3537217fea4133c9b74d544b5f3398f03d7c93cc156c007f074f76fc8e06fd1d"} Jul 18 14:04:15.619778 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:15.619723 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" podStartSLOduration=2.2343734570000002 podStartE2EDuration="2.619706176s" totalImagesPullingTime="385.332719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:04:15.617692145 +0000 UTC m=+1906.167256826" watchObservedRunningTime="2026-07-18 14:04:15.619706176 +0000 UTC m=+1906.169270857" Jul 18 14:04:16.607575 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:16.607539 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:21.383500 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:21.383466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:04:22.620109 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:22.620074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:29.298237 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.298034 2564 patch_prober.go:28] interesting pod/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:43410->10.133.0.56:15021: read: connection reset by peer" start-of-body= Jul 18 14:04:29.298237 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.298124 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:43410->10.133.0.56:15021: read: connection reset by peer" Jul 18 14:04:29.688732 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.688703 2564 generic.go:356] "Generic (PLEG): container finished" podID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerID="35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e" exitCode=0 Jul 18 14:04:29.688890 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.688774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" event={"ID":"94582a77-0c65-4c3a-bca1-f1108cfc7961","Type":"ContainerDied","Data":"35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e"} Jul 18 14:04:29.689218 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.689200 2564 scope.go:117] "RemoveContainer" containerID="35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e" Jul 18 14:04:29.690176 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.690146 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:04:29.690273 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.690214 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:04:29.690273 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:29.690245 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:04:30.696399 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:30.696353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" event={"ID":"94582a77-0c65-4c3a-bca1-f1108cfc7961","Type":"ContainerStarted","Data":"f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4"} Jul 18 14:04:31.176922 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:31.176835 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:31.177149 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:31.177123 2564 patch_prober.go:28] interesting pod/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 14:04:31.177218 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:31.177196 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 14:04:32.176996 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:32.176948 2564 patch_prober.go:28] interesting pod/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 14:04:32.177430 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:32.177018 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 14:04:33.176970 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:33.176925 2564 patch_prober.go:28] interesting pod/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 14:04:33.177157 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:33.176991 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 14:04:34.176722 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:34.176681 2564 patch_prober.go:28] interesting pod/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 14:04:34.176909 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:34.176739 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 14:04:35.179452 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:35.179420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:35.179919 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:35.179862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:35.180562 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:35.180542 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:49.134809 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.134777 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z"] Jul 18 14:04:49.135403 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.134987 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" podUID="b22a7439-efbd-45a0-a7a6-6043fd518238" containerName="maas-api" containerID="cri-o://777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa" gracePeriod=30 Jul 18 14:04:49.387332 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.387262 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:49.519406 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.519346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr9lx\" (UniqueName: \"kubernetes.io/projected/b22a7439-efbd-45a0-a7a6-6043fd518238-kube-api-access-xr9lx\") pod \"b22a7439-efbd-45a0-a7a6-6043fd518238\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " Jul 18 14:04:49.519406 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.519412 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls\") pod \"b22a7439-efbd-45a0-a7a6-6043fd518238\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " Jul 18 14:04:49.519664 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.519461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22a7439-efbd-45a0-a7a6-6043fd518238-openshift-service-ca\") pod \"b22a7439-efbd-45a0-a7a6-6043fd518238\" (UID: \"b22a7439-efbd-45a0-a7a6-6043fd518238\") " Jul 18 14:04:49.519908 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.519877 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22a7439-efbd-45a0-a7a6-6043fd518238-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b22a7439-efbd-45a0-a7a6-6043fd518238" (UID: "b22a7439-efbd-45a0-a7a6-6043fd518238"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:04:49.521689 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.521661 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22a7439-efbd-45a0-a7a6-6043fd518238-kube-api-access-xr9lx" (OuterVolumeSpecName: "kube-api-access-xr9lx") pod "b22a7439-efbd-45a0-a7a6-6043fd518238" (UID: "b22a7439-efbd-45a0-a7a6-6043fd518238"). InnerVolumeSpecName "kube-api-access-xr9lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:04:49.521778 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.521666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b22a7439-efbd-45a0-a7a6-6043fd518238" (UID: "b22a7439-efbd-45a0-a7a6-6043fd518238"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 14:04:49.620473 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.620433 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9lx\" (UniqueName: \"kubernetes.io/projected/b22a7439-efbd-45a0-a7a6-6043fd518238-kube-api-access-xr9lx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:49.620473 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.620466 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22a7439-efbd-45a0-a7a6-6043fd518238-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:49.620473 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.620475 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22a7439-efbd-45a0-a7a6-6043fd518238-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:49.803873 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.803831 2564 generic.go:356] "Generic (PLEG): container finished" podID="b22a7439-efbd-45a0-a7a6-6043fd518238" containerID="777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa" exitCode=0 Jul 18 14:04:49.804079 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.803880 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" Jul 18 14:04:49.804079 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.803908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" event={"ID":"b22a7439-efbd-45a0-a7a6-6043fd518238","Type":"ContainerDied","Data":"777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa"} Jul 18 14:04:49.804079 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.803962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z" event={"ID":"b22a7439-efbd-45a0-a7a6-6043fd518238","Type":"ContainerDied","Data":"3537217fea4133c9b74d544b5f3398f03d7c93cc156c007f074f76fc8e06fd1d"} Jul 18 14:04:49.804079 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.803987 2564 scope.go:117] "RemoveContainer" containerID="777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa" Jul 18 14:04:49.814574 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.814553 2564 scope.go:117] "RemoveContainer" containerID="777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa" Jul 18 14:04:49.814826 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:04:49.814806 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa\": container with ID starting with 777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa not found: ID does not exist" containerID="777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa" Jul 18 14:04:49.814874 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.814834 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa"} err="failed to get container status \"777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa\": rpc error: code = NotFound desc = could not find container \"777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa\": container with ID starting with 777eb7fa22ba427089966ec5b46197ba47b22f9ae6ebee88f57dbd09db1b59fa not found: ID does not exist" Jul 18 14:04:49.826173 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.826148 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z"] Jul 18 14:04:49.828618 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.828596 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ea60214d-5cdc78587b-h578z"] Jul 18 14:04:49.973465 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:49.973429 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b22a7439-efbd-45a0-a7a6-6043fd518238" path="/var/lib/kubelet/pods/b22a7439-efbd-45a0-a7a6-6043fd518238/volumes" Jul 18 14:04:51.990392 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:51.990352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:04:53.979279 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:53.979246 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm"] Jul 18 14:04:53.979860 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:53.979509 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" containerID="cri-o://f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4" gracePeriod=30 Jul 18 14:04:59.240580 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.240511 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:59.309810 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.309778 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-token\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.309985 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.309818 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-data\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.309985 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.309865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-envoy\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.309985 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.309916 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-podinfo\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.309985 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.309953 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-credential-socket\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.310196 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.309998 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v477j\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-kube-api-access-v477j\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.310196 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94582a77-0c65-4c3a-bca1-f1108cfc7961-istiod-ca-cert\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.310196 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310058 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-socket\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.310196 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310152 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-certs\") pod \"94582a77-0c65-4c3a-bca1-f1108cfc7961\" (UID: \"94582a77-0c65-4c3a-bca1-f1108cfc7961\") " Jul 18 14:04:59.310471 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310437 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:04:59.310533 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94582a77-0c65-4c3a-bca1-f1108cfc7961-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:04:59.310737 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310714 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:04:59.310844 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310754 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:04:59.310844 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.310782 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-data" (OuterVolumeSpecName: "istio-data") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:04:59.312088 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.312054 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:04:59.312218 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.312174 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 14:04:59.312462 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.312446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-token" (OuterVolumeSpecName: "istio-token") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:04:59.312557 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.312538 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-kube-api-access-v477j" (OuterVolumeSpecName: "kube-api-access-v477j") pod "94582a77-0c65-4c3a-bca1-f1108cfc7961" (UID: "94582a77-0c65-4c3a-bca1-f1108cfc7961"). InnerVolumeSpecName "kube-api-access-v477j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:04:59.411343 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411302 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411343 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411337 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411343 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411351 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411634 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411365 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411634 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411405 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94582a77-0c65-4c3a-bca1-f1108cfc7961-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411634 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411417 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411634 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411431 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v477j\" (UniqueName: \"kubernetes.io/projected/94582a77-0c65-4c3a-bca1-f1108cfc7961-kube-api-access-v477j\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411634 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411442 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94582a77-0c65-4c3a-bca1-f1108cfc7961-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.411634 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.411455 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94582a77-0c65-4c3a-bca1-f1108cfc7961-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:04:59.861511 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.861476 2564 generic.go:356] "Generic (PLEG): container finished" podID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerID="f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4" exitCode=0 Jul 18 14:04:59.861706 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.861536 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" Jul 18 14:04:59.861706 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.861565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" event={"ID":"94582a77-0c65-4c3a-bca1-f1108cfc7961","Type":"ContainerDied","Data":"f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4"} Jul 18 14:04:59.861706 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.861660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm" event={"ID":"94582a77-0c65-4c3a-bca1-f1108cfc7961","Type":"ContainerDied","Data":"120038eed72a8cdfdbb1d25b43d7ab2af09079d48a49d7f7f013d9e694e8ecad"} Jul 18 14:04:59.861706 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.861689 2564 scope.go:117] "RemoveContainer" containerID="f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4" Jul 18 14:04:59.871846 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.871827 2564 scope.go:117] "RemoveContainer" containerID="35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e" Jul 18 14:04:59.880244 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.880225 2564 scope.go:117] "RemoveContainer" containerID="f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4" Jul 18 14:04:59.880551 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:04:59.880529 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4\": container with ID starting with f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4 not found: ID does not exist" containerID="f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4" Jul 18 14:04:59.880606 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.880561 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4"} err="failed to get container status \"f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4\": rpc error: code = NotFound desc = could not find container \"f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4\": container with ID starting with f74f3f6808a9ff2e3edcd6158c28c675e9e085cbbef23eac1780f21a9aa0b0a4 not found: ID does not exist" Jul 18 14:04:59.880606 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.880580 2564 scope.go:117] "RemoveContainer" containerID="35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e" Jul 18 14:04:59.880832 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:04:59.880814 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e\": container with ID starting with 35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e not found: ID does not exist" containerID="35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e" Jul 18 14:04:59.880883 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.880838 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e"} err="failed to get container status \"35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e\": rpc error: code = NotFound desc = could not find container \"35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e\": container with ID starting with 35778f4f2e5b5e1cfd3dbdef2e7bcf19b2d0ba61af28bdf394c4db31fb665a5e not found: ID does not exist" Jul 18 14:04:59.884196 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.884173 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm"] Jul 18 14:04:59.890916 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.890890 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ea60214d-openshift-default-65656f5f4c-qfdfm"] Jul 18 14:04:59.972668 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:04:59.972632 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" path="/var/lib/kubelet/pods/94582a77-0c65-4c3a-bca1-f1108cfc7961/volumes" Jul 18 14:06:18.591939 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:18.591898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:06:29.786733 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.786693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:06:29.819623 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.819594 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v"] Jul 18 14:06:29.820016 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820001 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22a7439-efbd-45a0-a7a6-6043fd518238" containerName="maas-api" Jul 18 14:06:29.820061 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820020 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22a7439-efbd-45a0-a7a6-6043fd518238" containerName="maas-api" Jul 18 14:06:29.820061 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820031 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" Jul 18 14:06:29.820061 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820037 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" Jul 18 14:06:29.820061 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820052 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" Jul 18 14:06:29.820061 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820057 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" Jul 18 14:06:29.820215 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820134 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b22a7439-efbd-45a0-a7a6-6043fd518238" containerName="maas-api" Jul 18 14:06:29.820215 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820144 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" Jul 18 14:06:29.820215 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.820152 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="94582a77-0c65-4c3a-bca1-f1108cfc7961" containerName="istio-proxy" Jul 18 14:06:29.823423 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.823401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.825902 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.825875 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2d50f1fd-openshift-default-dockercfg-7ls2t\"" Jul 18 14:06:29.841790 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.841761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v"] Jul 18 14:06:29.927391 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-certs\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927599 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-data\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927599 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istiod-ca-cert\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927719 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-token\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927719 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6rt\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-kube-api-access-5z6rt\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927825 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-credential-socket\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927825 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-envoy\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927934 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-socket\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:29.927934 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:29.927884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-podinfo\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028594 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-certs\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028760 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-data\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028760 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istiod-ca-cert\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028760 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-token\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028899 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6rt\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-kube-api-access-5z6rt\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028899 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-credential-socket\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.028899 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-envoy\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029059 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-socket\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029059 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-podinfo\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029059 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.028977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-data\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029059 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.029010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-certs\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029270 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.029119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-credential-socket\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029347 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.029326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-socket\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.029777 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.029754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istiod-ca-cert\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.031194 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.031174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-envoy\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.031444 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.031424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-podinfo\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.037212 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.037150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-token\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.037286 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.037228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6rt\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-kube-api-access-5z6rt\") pod \"e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.134669 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.134637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:30.278645 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.278616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v"] Jul 18 14:06:30.279785 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:06:30.279755 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c6c565_c5fe_4ade_b56e_f2bf45b36d48.slice/crio-f14a55a82e769011770e8d42e0584e7d47eb601d1319d81d3b3825e2cc60af61 WatchSource:0}: Error finding container f14a55a82e769011770e8d42e0584e7d47eb601d1319d81d3b3825e2cc60af61: Status 404 returned error can't find the container with id f14a55a82e769011770e8d42e0584e7d47eb601d1319d81d3b3825e2cc60af61 Jul 18 14:06:30.281970 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.281933 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:06:30.282087 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.282020 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:06:30.282087 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.282065 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:06:30.365465 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.365421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" event={"ID":"28c6c565-c5fe-4ade-b56e-f2bf45b36d48","Type":"ContainerStarted","Data":"051728c20ae3eabf03b25db5de9524126c7d76f68c256de20f8a3b40e8d2b53a"} Jul 18 14:06:30.365649 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.365483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" event={"ID":"28c6c565-c5fe-4ade-b56e-f2bf45b36d48","Type":"ContainerStarted","Data":"f14a55a82e769011770e8d42e0584e7d47eb601d1319d81d3b3825e2cc60af61"} Jul 18 14:06:30.382746 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:30.382671 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podStartSLOduration=1.382649826 podStartE2EDuration="1.382649826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:06:30.382544203 +0000 UTC m=+2040.932108885" watchObservedRunningTime="2026-07-18 14:06:30.382649826 +0000 UTC m=+2040.932214510" Jul 18 14:06:31.135017 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:31.134986 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:31.139933 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:31.139907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:31.372116 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:31.372082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:31.373034 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:31.373013 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:38.589052 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:38.589012 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:06:46.610742 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:46.610690 2564 patch_prober.go:28] interesting pod/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:40306->10.133.0.58:15021: read: connection reset by peer" start-of-body= Jul 18 14:06:46.611107 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:46.610784 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:40306->10.133.0.58:15021: read: connection reset by peer" Jul 18 14:06:47.475905 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:47.475869 2564 generic.go:356] "Generic (PLEG): container finished" podID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerID="051728c20ae3eabf03b25db5de9524126c7d76f68c256de20f8a3b40e8d2b53a" exitCode=0 Jul 18 14:06:47.476082 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:47.475948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" event={"ID":"28c6c565-c5fe-4ade-b56e-f2bf45b36d48","Type":"ContainerDied","Data":"051728c20ae3eabf03b25db5de9524126c7d76f68c256de20f8a3b40e8d2b53a"} Jul 18 14:06:47.476412 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:47.476398 2564 scope.go:117] "RemoveContainer" containerID="051728c20ae3eabf03b25db5de9524126c7d76f68c256de20f8a3b40e8d2b53a" Jul 18 14:06:47.477385 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:47.477345 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:06:47.477459 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:47.477436 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:06:47.477498 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:47.477468 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:06:48.483234 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:48.483201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" event={"ID":"28c6c565-c5fe-4ade-b56e-f2bf45b36d48","Type":"ContainerStarted","Data":"0dcd15e7e204e8b44acbb6919968b2094e6de1ee6f2dee9d97ebf63014b228ae"} Jul 18 14:06:49.139399 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:49.135790 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:49.139399 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:49.135917 2564 patch_prober.go:28] interesting pod/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 18 14:06:49.139399 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:49.135962 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 18 14:06:50.136108 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:50.136072 2564 patch_prober.go:28] interesting pod/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 18 14:06:50.136724 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:50.136689 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 18 14:06:51.135322 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:51.135280 2564 patch_prober.go:28] interesting pod/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 18 14:06:51.135670 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:51.135636 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 18 14:06:52.135910 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:52.135814 2564 patch_prober.go:28] interesting pod/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 18 14:06:52.136510 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:52.136477 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 18 14:06:53.138808 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:53.138769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:53.139261 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:53.139069 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:53.139967 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:53.139949 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:06:57.583999 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:06:57.583961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:07:05.446721 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:05.446681 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v"] Jul 18 14:07:05.447311 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:05.446941 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" containerID="cri-o://0dcd15e7e204e8b44acbb6919968b2094e6de1ee6f2dee9d97ebf63014b228ae" gracePeriod=30 Jul 18 14:07:08.142217 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:08.142182 2564 patch_prober.go:28] interesting pod/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 14:07:08.142615 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:08.142271 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 14:07:10.609960 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.609925 2564 generic.go:356] "Generic (PLEG): container finished" podID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerID="0dcd15e7e204e8b44acbb6919968b2094e6de1ee6f2dee9d97ebf63014b228ae" exitCode=0 Jul 18 14:07:10.610369 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.609995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" event={"ID":"28c6c565-c5fe-4ade-b56e-f2bf45b36d48","Type":"ContainerDied","Data":"0dcd15e7e204e8b44acbb6919968b2094e6de1ee6f2dee9d97ebf63014b228ae"} Jul 18 14:07:10.610369 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.610046 2564 scope.go:117] "RemoveContainer" containerID="051728c20ae3eabf03b25db5de9524126c7d76f68c256de20f8a3b40e8d2b53a" Jul 18 14:07:10.704896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.704870 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:07:10.746811 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746725 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-credential-socket\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.746811 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-token\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.746811 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746806 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-certs\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istiod-ca-cert\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746858 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-socket\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746891 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-envoy\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.746944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-podinfo\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747010 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-data\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z6rt\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-kube-api-access-5z6rt\") pod \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\" (UID: \"28c6c565-c5fe-4ade-b56e-f2bf45b36d48\") " Jul 18 14:07:10.747090 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747050 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:07:10.747496 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747190 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:07:10.747496 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:07:10.747496 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747421 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.747655 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:07:10.747655 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747580 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.747655 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.747597 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.748170 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.748142 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-data" (OuterVolumeSpecName: "istio-data") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:07:10.749290 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.749266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 14:07:10.749290 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.749275 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-kube-api-access-5z6rt" (OuterVolumeSpecName: "kube-api-access-5z6rt") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "kube-api-access-5z6rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:07:10.749446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.749312 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:07:10.749566 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.749551 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-token" (OuterVolumeSpecName: "istio-token") pod "28c6c565-c5fe-4ade-b56e-f2bf45b36d48" (UID: "28c6c565-c5fe-4ade-b56e-f2bf45b36d48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:07:10.848586 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.848551 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.848586 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.848583 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6rt\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-kube-api-access-5z6rt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.848586 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.848593 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.848840 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.848601 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.848840 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.848609 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:10.848840 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:10.848617 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28c6c565-c5fe-4ade-b56e-f2bf45b36d48-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:11.616466 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:11.616438 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" Jul 18 14:07:11.616976 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:11.616438 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v" event={"ID":"28c6c565-c5fe-4ade-b56e-f2bf45b36d48","Type":"ContainerDied","Data":"f14a55a82e769011770e8d42e0584e7d47eb601d1319d81d3b3825e2cc60af61"} Jul 18 14:07:11.616976 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:11.616580 2564 scope.go:117] "RemoveContainer" containerID="0dcd15e7e204e8b44acbb6919968b2094e6de1ee6f2dee9d97ebf63014b228ae" Jul 18 14:07:11.638355 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:11.638325 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v"] Jul 18 14:07:11.644798 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:11.644768 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2d50f1fd-openshift-default-6477558c9-t5m7v"] Jul 18 14:07:11.974107 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:11.974075 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" path="/var/lib/kubelet/pods/28c6c565-c5fe-4ade-b56e-f2bf45b36d48/volumes" Jul 18 14:07:29.877957 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:29.877919 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:07:39.378747 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:39.378709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:07:51.491525 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:51.491484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:07:54.341233 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341195 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g"] Jul 18 14:07:54.341629 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341614 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" Jul 18 14:07:54.341629 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341627 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" Jul 18 14:07:54.341702 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341639 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" Jul 18 14:07:54.341702 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341645 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" Jul 18 14:07:54.341768 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341714 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" Jul 18 14:07:54.341768 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.341723 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c6c565-c5fe-4ade-b56e-f2bf45b36d48" containerName="istio-proxy" Jul 18 14:07:54.344884 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.344865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:54.347393 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.347352 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:07:54.352905 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.352711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g"] Jul 18 14:07:54.455220 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.455183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c885b696-566f-41fb-822e-21f32e2b3cb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g\" (UID: \"c885b696-566f-41fb-822e-21f32e2b3cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:54.556400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.556337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c885b696-566f-41fb-822e-21f32e2b3cb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g\" (UID: \"c885b696-566f-41fb-822e-21f32e2b3cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:54.557053 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.557033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c885b696-566f-41fb-822e-21f32e2b3cb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g\" (UID: \"c885b696-566f-41fb-822e-21f32e2b3cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:54.657505 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.657414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:54.793065 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.793034 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g"] Jul 18 14:07:54.795156 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:07:54.795126 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc885b696_566f_41fb_822e_21f32e2b3cb7.slice/crio-c01f191b1c415339901388eabfc3ac8736b078355609ba036cb63d5bc9a70ed5 WatchSource:0}: Error finding container c01f191b1c415339901388eabfc3ac8736b078355609ba036cb63d5bc9a70ed5: Status 404 returned error can't find the container with id c01f191b1c415339901388eabfc3ac8736b078355609ba036cb63d5bc9a70ed5 Jul 18 14:07:54.856660 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:54.856624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" event={"ID":"c885b696-566f-41fb-822e-21f32e2b3cb7","Type":"ContainerStarted","Data":"c01f191b1c415339901388eabfc3ac8736b078355609ba036cb63d5bc9a70ed5"} Jul 18 14:07:55.864049 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:55.863951 2564 generic.go:356] "Generic (PLEG): container finished" podID="c885b696-566f-41fb-822e-21f32e2b3cb7" containerID="ce69aa6f57309b89bc280095f87d5b0d488418aa8befce49b9e442f168de4652" exitCode=0 Jul 18 14:07:55.864462 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:55.864042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" event={"ID":"c885b696-566f-41fb-822e-21f32e2b3cb7","Type":"ContainerDied","Data":"ce69aa6f57309b89bc280095f87d5b0d488418aa8befce49b9e442f168de4652"} Jul 18 14:07:57.009182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.009151 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:57.078803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.078765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c885b696-566f-41fb-822e-21f32e2b3cb7-maas-api-service-ca\") pod \"c885b696-566f-41fb-822e-21f32e2b3cb7\" (UID: \"c885b696-566f-41fb-822e-21f32e2b3cb7\") " Jul 18 14:07:57.079033 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.079008 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c885b696-566f-41fb-822e-21f32e2b3cb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c885b696-566f-41fb-822e-21f32e2b3cb7" (UID: "c885b696-566f-41fb-822e-21f32e2b3cb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:07:57.180110 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.180013 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c885b696-566f-41fb-822e-21f32e2b3cb7-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:07:57.876691 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.876665 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" Jul 18 14:07:57.876873 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.876660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g" event={"ID":"c885b696-566f-41fb-822e-21f32e2b3cb7","Type":"ContainerDied","Data":"c01f191b1c415339901388eabfc3ac8736b078355609ba036cb63d5bc9a70ed5"} Jul 18 14:07:57.876873 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:07:57.876798 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c01f191b1c415339901388eabfc3ac8736b078355609ba036cb63d5bc9a70ed5" Jul 18 14:08:08.187659 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:08:08.187626 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:08:14.409616 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:08:14.409575 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g"] Jul 18 14:08:14.411797 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:08:14.411771 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ffbfeb3b-bc84g"] Jul 18 14:08:15.973473 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:08:15.973441 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c885b696-566f-41fb-822e-21f32e2b3cb7" path="/var/lib/kubelet/pods/c885b696-566f-41fb-822e-21f32e2b3cb7/volumes" Jul 18 14:08:18.593877 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:08:18.593843 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:09:07.992342 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:09:07.992301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:09:19.386488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:09:19.386450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:10:06.294943 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.294854 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:10:06.819336 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.819291 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld"] Jul 18 14:10:06.819994 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.819971 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c885b696-566f-41fb-822e-21f32e2b3cb7" containerName="revoke-keys" Jul 18 14:10:06.820066 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.819998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c885b696-566f-41fb-822e-21f32e2b3cb7" containerName="revoke-keys" Jul 18 14:10:06.820124 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.820101 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c885b696-566f-41fb-822e-21f32e2b3cb7" containerName="revoke-keys" Jul 18 14:10:06.823666 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.823645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.826385 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.826345 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-45d555-openshift-default-dockercfg-crnd9\"" Jul 18 14:10:06.832022 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.831993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld"] Jul 18 14:10:06.868543 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-socket\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.868543 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hvs\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-kube-api-access-v8hvs\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.868806 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-credential-socket\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.868806 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-data\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.868806 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-token\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.868806 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-podinfo\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.868806 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-certs\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.869045 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istiod-ca-cert\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.869045 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.868934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-envoy\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969531 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-socket\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969531 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hvs\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-kube-api-access-v8hvs\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-credential-socket\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-data\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-token\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-podinfo\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-certs\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istiod-ca-cert\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.969803 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-envoy\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.970130 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.969960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-socket\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.970130 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.970009 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-credential-socket\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.970130 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.970029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-data\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.970230 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.970183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-certs\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.970553 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.970524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istiod-ca-cert\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.972111 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.972078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-envoy\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.972225 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.972078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-podinfo\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.982456 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.982430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-token\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:06.982561 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:06.982443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hvs\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-kube-api-access-v8hvs\") pod \"e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:07.136074 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.135994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:07.283019 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.282989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld"] Jul 18 14:10:07.284431 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:10:07.284403 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358ad5d5_1c97_4aa6_a36c_859cc534f1f4.slice/crio-024b2b356ca6920fa47f00c59943c61bcee41c992828fca6d9763dabecd3a1bb WatchSource:0}: Error finding container 024b2b356ca6920fa47f00c59943c61bcee41c992828fca6d9763dabecd3a1bb: Status 404 returned error can't find the container with id 024b2b356ca6920fa47f00c59943c61bcee41c992828fca6d9763dabecd3a1bb Jul 18 14:10:07.286520 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.286483 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:10:07.286596 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.286559 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:10:07.286596 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.286587 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:10:07.586345 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.586304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" event={"ID":"358ad5d5-1c97-4aa6-a36c-859cc534f1f4","Type":"ContainerStarted","Data":"9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff"} Jul 18 14:10:07.586819 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.586364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" event={"ID":"358ad5d5-1c97-4aa6-a36c-859cc534f1f4","Type":"ContainerStarted","Data":"024b2b356ca6920fa47f00c59943c61bcee41c992828fca6d9763dabecd3a1bb"} Jul 18 14:10:07.610366 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:07.610310 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podStartSLOduration=1.610292117 podStartE2EDuration="1.610292117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:10:07.607482831 +0000 UTC m=+2258.157047512" watchObservedRunningTime="2026-07-18 14:10:07.610292117 +0000 UTC m=+2258.159856798" Jul 18 14:10:08.136507 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:08.136475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:08.141808 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:08.141786 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:08.592104 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:08.592074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:08.593103 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:08.593082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:17.390791 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:17.390752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:10:24.635579 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:24.635531 2564 patch_prober.go:28] interesting pod/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 18 14:10:24.636230 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:24.635629 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 18 14:10:25.698569 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:25.698535 2564 generic.go:356] "Generic (PLEG): container finished" podID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerID="9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff" exitCode=0 Jul 18 14:10:25.698986 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:25.698609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" event={"ID":"358ad5d5-1c97-4aa6-a36c-859cc534f1f4","Type":"ContainerDied","Data":"9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff"} Jul 18 14:10:25.699035 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:25.699025 2564 scope.go:117] "RemoveContainer" containerID="9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff" Jul 18 14:10:25.700037 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:25.700010 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:10:25.700103 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:25.700078 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:10:25.700139 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:25.700127 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:10:26.706075 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:26.706033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" event={"ID":"358ad5d5-1c97-4aa6-a36c-859cc534f1f4","Type":"ContainerStarted","Data":"7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2"} Jul 18 14:10:27.137296 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:27.137197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:27.137703 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:27.137611 2564 patch_prober.go:28] interesting pod/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 18 14:10:27.137908 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:27.137882 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 18 14:10:27.784019 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:27.783983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:10:28.137729 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:28.137171 2564 patch_prober.go:28] interesting pod/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 18 14:10:28.137729 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:28.137237 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 18 14:10:29.138398 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:29.137509 2564 patch_prober.go:28] interesting pod/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 18 14:10:29.138398 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:29.137582 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 18 14:10:30.136728 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:30.136685 2564 patch_prober.go:28] interesting pod/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 18 14:10:30.137043 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:30.137011 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 18 14:10:31.140338 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:31.140307 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:31.140767 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:31.140624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:31.141398 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:31.141350 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:10:55.288361 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:10:55.288322 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:11:08.141337 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.141305 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4"] Jul 18 14:11:08.145046 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.145023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:08.147540 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.147518 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:11:08.152272 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.152244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4"] Jul 18 14:11:08.246445 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.246400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049726e3-2d9b-4a9a-bfea-3211643f7d45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4\" (UID: \"049726e3-2d9b-4a9a-bfea-3211643f7d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:08.347471 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.347431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049726e3-2d9b-4a9a-bfea-3211643f7d45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4\" (UID: \"049726e3-2d9b-4a9a-bfea-3211643f7d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:08.348069 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.348047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049726e3-2d9b-4a9a-bfea-3211643f7d45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4\" (UID: \"049726e3-2d9b-4a9a-bfea-3211643f7d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:08.458173 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.458143 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:08.597182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.597150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4"] Jul 18 14:11:08.598605 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:11:08.598574 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049726e3_2d9b_4a9a_bfea_3211643f7d45.slice/crio-ddda6cd343aa1fe8d7e5f0883141e4a89e0c6c1ad15dd3b540a42f76701dce15 WatchSource:0}: Error finding container ddda6cd343aa1fe8d7e5f0883141e4a89e0c6c1ad15dd3b540a42f76701dce15: Status 404 returned error can't find the container with id ddda6cd343aa1fe8d7e5f0883141e4a89e0c6c1ad15dd3b540a42f76701dce15 Jul 18 14:11:08.941150 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.941112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" event={"ID":"049726e3-2d9b-4a9a-bfea-3211643f7d45","Type":"ContainerStarted","Data":"90e3813c20ffc97a6a0dddc0632f85e8604f21c78c99dd6017b44c5f837ad08d"} Jul 18 14:11:08.941292 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.941213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" event={"ID":"049726e3-2d9b-4a9a-bfea-3211643f7d45","Type":"ContainerStarted","Data":"ddda6cd343aa1fe8d7e5f0883141e4a89e0c6c1ad15dd3b540a42f76701dce15"} Jul 18 14:11:08.954896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:08.954845 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" podStartSLOduration=0.95482946 podStartE2EDuration="954.82946ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:11:08.95422633 +0000 UTC m=+2319.503791013" watchObservedRunningTime="2026-07-18 14:11:08.95482946 +0000 UTC m=+2319.504394141" Jul 18 14:11:09.948725 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:09.948688 2564 generic.go:356] "Generic (PLEG): container finished" podID="049726e3-2d9b-4a9a-bfea-3211643f7d45" containerID="90e3813c20ffc97a6a0dddc0632f85e8604f21c78c99dd6017b44c5f837ad08d" exitCode=0 Jul 18 14:11:09.949118 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:09.948771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" event={"ID":"049726e3-2d9b-4a9a-bfea-3211643f7d45","Type":"ContainerDied","Data":"90e3813c20ffc97a6a0dddc0632f85e8604f21c78c99dd6017b44c5f837ad08d"} Jul 18 14:11:11.088562 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.088530 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:11.176632 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.176601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049726e3-2d9b-4a9a-bfea-3211643f7d45-maas-api-service-ca\") pod \"049726e3-2d9b-4a9a-bfea-3211643f7d45\" (UID: \"049726e3-2d9b-4a9a-bfea-3211643f7d45\") " Jul 18 14:11:11.177010 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.176985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049726e3-2d9b-4a9a-bfea-3211643f7d45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "049726e3-2d9b-4a9a-bfea-3211643f7d45" (UID: "049726e3-2d9b-4a9a-bfea-3211643f7d45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:11:11.278037 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.277938 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049726e3-2d9b-4a9a-bfea-3211643f7d45-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:11.962166 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.962134 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" Jul 18 14:11:11.962356 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.962172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4" event={"ID":"049726e3-2d9b-4a9a-bfea-3211643f7d45","Type":"ContainerDied","Data":"ddda6cd343aa1fe8d7e5f0883141e4a89e0c6c1ad15dd3b540a42f76701dce15"} Jul 18 14:11:11.962356 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:11.962227 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddda6cd343aa1fe8d7e5f0883141e4a89e0c6c1ad15dd3b540a42f76701dce15" Jul 18 14:11:13.789545 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:13.789506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:11:23.190189 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:23.190102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:11:23.210406 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:23.210347 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4"] Jul 18 14:11:23.212993 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:23.212964 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5f2a1-2jxv4"] Jul 18 14:11:23.973484 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:23.973451 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049726e3-2d9b-4a9a-bfea-3211643f7d45" path="/var/lib/kubelet/pods/049726e3-2d9b-4a9a-bfea-3211643f7d45/volumes" Jul 18 14:11:29.951073 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.951033 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx"] Jul 18 14:11:29.951564 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.951457 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049726e3-2d9b-4a9a-bfea-3211643f7d45" containerName="revoke-keys" Jul 18 14:11:29.951564 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.951470 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="049726e3-2d9b-4a9a-bfea-3211643f7d45" containerName="revoke-keys" Jul 18 14:11:29.951564 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.951546 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="049726e3-2d9b-4a9a-bfea-3211643f7d45" containerName="revoke-keys" Jul 18 14:11:29.955766 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.955748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:29.958396 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.958348 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:11:29.961407 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:29.961362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx"] Jul 18 14:11:30.050149 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:30.050113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e119d51d-599d-44ea-a3bb-c03b82bec938-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-45d555-956cx\" (UID: \"e119d51d-599d-44ea-a3bb-c03b82bec938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:30.150783 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:30.150737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e119d51d-599d-44ea-a3bb-c03b82bec938-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-45d555-956cx\" (UID: \"e119d51d-599d-44ea-a3bb-c03b82bec938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:30.151403 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:30.151354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e119d51d-599d-44ea-a3bb-c03b82bec938-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-45d555-956cx\" (UID: \"e119d51d-599d-44ea-a3bb-c03b82bec938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:30.268886 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:30.268790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:30.408025 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:30.407981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx"] Jul 18 14:11:30.410542 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:11:30.410512 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode119d51d_599d_44ea_a3bb_c03b82bec938.slice/crio-40525b071f4051a788902257bb3db91642c1511aa849e491203c8625c67fa44c WatchSource:0}: Error finding container 40525b071f4051a788902257bb3db91642c1511aa849e491203c8625c67fa44c: Status 404 returned error can't find the container with id 40525b071f4051a788902257bb3db91642c1511aa849e491203c8625c67fa44c Jul 18 14:11:31.066932 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:31.066899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" event={"ID":"e119d51d-599d-44ea-a3bb-c03b82bec938","Type":"ContainerStarted","Data":"e3be589b942f45b7236a7153b48ffce52e0859b97d5f71ba197c7be0a68a3fce"} Jul 18 14:11:31.067365 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:31.066958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" event={"ID":"e119d51d-599d-44ea-a3bb-c03b82bec938","Type":"ContainerStarted","Data":"40525b071f4051a788902257bb3db91642c1511aa849e491203c8625c67fa44c"} Jul 18 14:11:31.083482 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:31.083418 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" podStartSLOduration=2.083401356 podStartE2EDuration="2.083401356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:11:31.080680202 +0000 UTC m=+2341.630244883" watchObservedRunningTime="2026-07-18 14:11:31.083401356 +0000 UTC m=+2341.632966039" Jul 18 14:11:32.073941 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:32.073897 2564 generic.go:356] "Generic (PLEG): container finished" podID="e119d51d-599d-44ea-a3bb-c03b82bec938" containerID="e3be589b942f45b7236a7153b48ffce52e0859b97d5f71ba197c7be0a68a3fce" exitCode=0 Jul 18 14:11:32.074358 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:32.073980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" event={"ID":"e119d51d-599d-44ea-a3bb-c03b82bec938","Type":"ContainerDied","Data":"e3be589b942f45b7236a7153b48ffce52e0859b97d5f71ba197c7be0a68a3fce"} Jul 18 14:11:33.220058 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:33.220034 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:33.278538 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:33.278502 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e119d51d-599d-44ea-a3bb-c03b82bec938-maas-api-service-ca\") pod \"e119d51d-599d-44ea-a3bb-c03b82bec938\" (UID: \"e119d51d-599d-44ea-a3bb-c03b82bec938\") " Jul 18 14:11:33.278901 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:33.278872 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e119d51d-599d-44ea-a3bb-c03b82bec938-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e119d51d-599d-44ea-a3bb-c03b82bec938" (UID: "e119d51d-599d-44ea-a3bb-c03b82bec938"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:11:33.380074 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:33.379979 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e119d51d-599d-44ea-a3bb-c03b82bec938-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:34.088266 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:34.088228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" event={"ID":"e119d51d-599d-44ea-a3bb-c03b82bec938","Type":"ContainerDied","Data":"40525b071f4051a788902257bb3db91642c1511aa849e491203c8625c67fa44c"} Jul 18 14:11:34.088454 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:34.088285 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40525b071f4051a788902257bb3db91642c1511aa849e491203c8625c67fa44c" Jul 18 14:11:34.088454 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:34.088237 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx" Jul 18 14:11:43.083445 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:43.083409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:11:45.014474 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:45.014433 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx"] Jul 18 14:11:45.018471 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:45.018439 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-45d555-956cx"] Jul 18 14:11:45.973350 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:45.973314 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e119d51d-599d-44ea-a3bb-c03b82bec938" path="/var/lib/kubelet/pods/e119d51d-599d-44ea-a3bb-c03b82bec938/volumes" Jul 18 14:11:51.193149 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:51.193115 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld"] Jul 18 14:11:51.193612 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:51.193389 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" containerID="cri-o://7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2" gracePeriod=30 Jul 18 14:11:56.458555 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.458487 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:11:56.592740 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592705 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-certs\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.592740 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istiod-ca-cert\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.592971 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592769 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-data\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.592971 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592789 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-token\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.592971 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-envoy\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.592971 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-credential-socket\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.592971 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.592969 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8hvs\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-kube-api-access-v8hvs\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.593205 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593023 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-socket\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.593205 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593053 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-podinfo\") pod \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\" (UID: \"358ad5d5-1c97-4aa6-a36c-859cc534f1f4\") " Jul 18 14:11:56.593205 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:11:56.593370 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:11:56.593370 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593247 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:11:56.593541 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593462 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.593541 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593480 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.593541 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593480 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:11:56.593541 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593493 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.593745 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.593724 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:11:56.595292 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.595263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:11:56.595447 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.595269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 14:11:56.595535 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.595513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:11:56.595583 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.595542 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-kube-api-access-v8hvs" (OuterVolumeSpecName: "kube-api-access-v8hvs") pod "358ad5d5-1c97-4aa6-a36c-859cc534f1f4" (UID: "358ad5d5-1c97-4aa6-a36c-859cc534f1f4"). InnerVolumeSpecName "kube-api-access-v8hvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:11:56.694982 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.694933 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.694982 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.694982 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.695222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.694996 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.695222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.695019 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hvs\" (UniqueName: \"kubernetes.io/projected/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-kube-api-access-v8hvs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.695222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.695035 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:56.695222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:56.695049 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358ad5d5-1c97-4aa6-a36c-859cc534f1f4-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:11:57.221443 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.221406 2564 generic.go:356] "Generic (PLEG): container finished" podID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerID="7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2" exitCode=0 Jul 18 14:11:57.221620 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.221470 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" Jul 18 14:11:57.221620 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.221475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" event={"ID":"358ad5d5-1c97-4aa6-a36c-859cc534f1f4","Type":"ContainerDied","Data":"7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2"} Jul 18 14:11:57.221620 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.221592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld" event={"ID":"358ad5d5-1c97-4aa6-a36c-859cc534f1f4","Type":"ContainerDied","Data":"024b2b356ca6920fa47f00c59943c61bcee41c992828fca6d9763dabecd3a1bb"} Jul 18 14:11:57.221620 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.221613 2564 scope.go:117] "RemoveContainer" containerID="7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2" Jul 18 14:11:57.231330 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.231309 2564 scope.go:117] "RemoveContainer" containerID="9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff" Jul 18 14:11:57.240222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.240196 2564 scope.go:117] "RemoveContainer" containerID="7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2" Jul 18 14:11:57.240610 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:11:57.240578 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2\": container with ID starting with 7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2 not found: ID does not exist" containerID="7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2" Jul 18 14:11:57.240735 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.240623 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2"} err="failed to get container status \"7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2\": rpc error: code = NotFound desc = could not find container \"7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2\": container with ID starting with 7a821440e6584821fafd062d1c682ec57921ad8cec64a566db5d8f8ed69bb2d2 not found: ID does not exist" Jul 18 14:11:57.240735 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.240658 2564 scope.go:117] "RemoveContainer" containerID="9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff" Jul 18 14:11:57.241002 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:11:57.240977 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff\": container with ID starting with 9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff not found: ID does not exist" containerID="9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff" Jul 18 14:11:57.241173 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.241135 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff"} err="failed to get container status \"9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff\": rpc error: code = NotFound desc = could not find container \"9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff\": container with ID starting with 9343994e63120501c3dd48dd26c34b41223c276c807137097c84a45bbcf418ff not found: ID does not exist" Jul 18 14:11:57.243466 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.243446 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld"] Jul 18 14:11:57.247262 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.247223 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-45d555-openshift-default-77f9b64795-g6bld"] Jul 18 14:11:57.973532 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:11:57.973499 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" path="/var/lib/kubelet/pods/358ad5d5-1c97-4aa6-a36c-859cc534f1f4/volumes" Jul 18 14:12:00.490805 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.490764 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd"] Jul 18 14:12:00.491488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491466 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" Jul 18 14:12:00.491540 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491491 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" Jul 18 14:12:00.491540 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491508 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" Jul 18 14:12:00.491540 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491517 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" Jul 18 14:12:00.491540 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491529 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e119d51d-599d-44ea-a3bb-c03b82bec938" containerName="revoke-keys" Jul 18 14:12:00.491540 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491538 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e119d51d-599d-44ea-a3bb-c03b82bec938" containerName="revoke-keys" Jul 18 14:12:00.491701 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491646 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" Jul 18 14:12:00.491701 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.491660 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e119d51d-599d-44ea-a3bb-c03b82bec938" containerName="revoke-keys" Jul 18 14:12:00.496483 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.496461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.499035 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.499018 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6ee9e9-openshift-default-dockercfg-r6t29\"" Jul 18 14:12:00.503149 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.503094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd"] Jul 18 14:12:00.633589 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-token\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633766 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276fp\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-kube-api-access-276fp\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633766 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-socket\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633766 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istiod-ca-cert\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633766 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-envoy\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633911 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-podinfo\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633911 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633812 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-data\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633911 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-credential-socket\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.633911 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.633909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-certs\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.734903 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.734863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-token\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.734903 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.734907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276fp\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-kube-api-access-276fp\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-socket\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istiod-ca-cert\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735414 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-envoy\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735488 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-podinfo\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735563 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-data\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735563 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-credential-socket\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.735674 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-socket\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.736082 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.735578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-certs\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.736270 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.736243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istiod-ca-cert\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.736361 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.736313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-data\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.736509 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.736486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-certs\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.736772 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.736749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-credential-socket\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.744480 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.744309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-envoy\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.744480 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.744449 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-podinfo\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.746596 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.746570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-token\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.746909 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.746887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-276fp\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-kube-api-access-276fp\") pod \"e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.808501 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.808460 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:00.961523 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.961488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd"] Jul 18 14:12:00.962650 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:12:00.962619 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d945bfb_61e1_47a7_a4ff_57aa7d2b5c05.slice/crio-06a84ace7d0cb0a973d660c80800620c8a25fbd406de801e9ded79fe8f5d4d96 WatchSource:0}: Error finding container 06a84ace7d0cb0a973d660c80800620c8a25fbd406de801e9ded79fe8f5d4d96: Status 404 returned error can't find the container with id 06a84ace7d0cb0a973d660c80800620c8a25fbd406de801e9ded79fe8f5d4d96 Jul 18 14:12:00.964737 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.964703 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:12:00.964849 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.964773 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:12:00.964849 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:00.964816 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:12:01.252824 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:01.252721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" event={"ID":"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05","Type":"ContainerStarted","Data":"3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1"} Jul 18 14:12:01.252824 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:01.252779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" event={"ID":"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05","Type":"ContainerStarted","Data":"06a84ace7d0cb0a973d660c80800620c8a25fbd406de801e9ded79fe8f5d4d96"} Jul 18 14:12:01.275840 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:01.275785 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podStartSLOduration=1.27576728 podStartE2EDuration="1.27576728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:12:01.272537938 +0000 UTC m=+2371.822102619" watchObservedRunningTime="2026-07-18 14:12:01.27576728 +0000 UTC m=+2371.825331961" Jul 18 14:12:01.809367 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:01.809325 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:02.814656 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:02.814625 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:03.263440 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:03.263405 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:03.264532 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:03.264512 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:03.493127 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:03.493086 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:12:15.482955 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.482910 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c"] Jul 18 14:12:15.483592 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.483571 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="358ad5d5-1c97-4aa6-a36c-859cc534f1f4" containerName="istio-proxy" Jul 18 14:12:15.486785 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.486761 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:15.489239 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.489216 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:12:15.494065 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.494041 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c"] Jul 18 14:12:15.580042 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.579997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c\" (UID: \"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:15.680881 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.680842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c\" (UID: \"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:15.681529 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.681507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c\" (UID: \"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:15.801560 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.801468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:15.947093 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.947017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c"] Jul 18 14:12:15.947495 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:12:15.947469 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a62fc0_b2d3_4c60_81ab_e29c9594e3eb.slice/crio-b6c142ec6602da01cfe06c0f72cbffbfac55c7ae6d7f9bead21bdbb2b87a7887 WatchSource:0}: Error finding container b6c142ec6602da01cfe06c0f72cbffbfac55c7ae6d7f9bead21bdbb2b87a7887: Status 404 returned error can't find the container with id b6c142ec6602da01cfe06c0f72cbffbfac55c7ae6d7f9bead21bdbb2b87a7887 Jul 18 14:12:15.986978 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:15.986948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:12:16.347128 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:16.347092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" event={"ID":"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb","Type":"ContainerStarted","Data":"4aa3d88a8146cf3d83ee48db2d830fe66111863f8d839b12046ee9944049cd45"} Jul 18 14:12:16.347334 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:16.347153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" event={"ID":"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb","Type":"ContainerStarted","Data":"b6c142ec6602da01cfe06c0f72cbffbfac55c7ae6d7f9bead21bdbb2b87a7887"} Jul 18 14:12:16.367194 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:16.367126 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" podStartSLOduration=1.3671036 podStartE2EDuration="1.3671036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:12:16.362392527 +0000 UTC m=+2386.911957212" watchObservedRunningTime="2026-07-18 14:12:16.3671036 +0000 UTC m=+2386.916668283" Jul 18 14:12:17.354502 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:17.354464 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb" containerID="4aa3d88a8146cf3d83ee48db2d830fe66111863f8d839b12046ee9944049cd45" exitCode=0 Jul 18 14:12:17.354912 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:17.354554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" event={"ID":"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb","Type":"ContainerDied","Data":"4aa3d88a8146cf3d83ee48db2d830fe66111863f8d839b12046ee9944049cd45"} Jul 18 14:12:18.505078 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:18.505051 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:18.608141 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:18.608106 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb-maas-api-service-ca\") pod \"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb\" (UID: \"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb\") " Jul 18 14:12:18.608486 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:18.608465 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb" (UID: "f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:12:18.709581 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:18.709545 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:12:18.716664 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:18.716627 2564 patch_prober.go:28] interesting pod/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:43798->10.133.0.63:15021: read: connection reset by peer" start-of-body= Jul 18 14:12:18.716822 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:18.716709 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:43798->10.133.0.63:15021: read: connection reset by peer" Jul 18 14:12:19.367185 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.367151 2564 generic.go:356] "Generic (PLEG): container finished" podID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerID="3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1" exitCode=0 Jul 18 14:12:19.367387 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.367227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" event={"ID":"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05","Type":"ContainerDied","Data":"3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1"} Jul 18 14:12:19.367721 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.367696 2564 scope.go:117] "RemoveContainer" containerID="3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1" Jul 18 14:12:19.368711 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.368690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" event={"ID":"f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb","Type":"ContainerDied","Data":"b6c142ec6602da01cfe06c0f72cbffbfac55c7ae6d7f9bead21bdbb2b87a7887"} Jul 18 14:12:19.368813 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.368714 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c" Jul 18 14:12:19.368813 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.368738 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6c142ec6602da01cfe06c0f72cbffbfac55c7ae6d7f9bead21bdbb2b87a7887" Jul 18 14:12:19.368921 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.368806 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:12:19.368921 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.368895 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:12:19.369019 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:19.368928 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:12:20.376891 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:20.376856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" event={"ID":"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05","Type":"ContainerStarted","Data":"a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2"} Jul 18 14:12:20.808642 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:20.808605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:20.808865 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:20.808845 2564 patch_prober.go:28] interesting pod/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 18 14:12:20.808953 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:20.808900 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 18 14:12:21.809677 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:21.809636 2564 patch_prober.go:28] interesting pod/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 18 14:12:21.810151 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:21.809696 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 18 14:12:22.809794 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:22.809754 2564 patch_prober.go:28] interesting pod/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 18 14:12:22.810269 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:22.809821 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 18 14:12:23.809600 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:23.809560 2564 patch_prober.go:28] interesting pod/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 18 14:12:23.809786 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:23.809619 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 18 14:12:24.812962 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:24.812927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:24.813454 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:24.813396 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:24.814301 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:24.814279 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:12:35.556047 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:35.556008 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c"] Jul 18 14:12:35.558566 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:35.558538 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-31be2e-kzv7c"] Jul 18 14:12:35.973725 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:35.973686 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb" path="/var/lib/kubelet/pods/f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb/volumes" Jul 18 14:12:37.488499 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:37.488462 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:12:42.327260 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.327226 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn"] Jul 18 14:12:42.327715 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.327637 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb" containerName="revoke-keys" Jul 18 14:12:42.327715 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.327649 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb" containerName="revoke-keys" Jul 18 14:12:42.327715 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.327714 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a62fc0-b2d3-4c60-81ab-e29c9594e3eb" containerName="revoke-keys" Jul 18 14:12:42.330814 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.330796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:42.333442 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.333413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:12:42.338420 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.338392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn"] Jul 18 14:12:42.439597 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.439562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59354e9-f17b-4507-b5c2-8be7b15bb7d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn\" (UID: \"b59354e9-f17b-4507-b5c2-8be7b15bb7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:42.541011 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.540970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59354e9-f17b-4507-b5c2-8be7b15bb7d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn\" (UID: \"b59354e9-f17b-4507-b5c2-8be7b15bb7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:42.541675 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.541652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59354e9-f17b-4507-b5c2-8be7b15bb7d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn\" (UID: \"b59354e9-f17b-4507-b5c2-8be7b15bb7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:42.643269 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.643178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:42.782849 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:42.782815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn"] Jul 18 14:12:42.783759 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:12:42.783736 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59354e9_f17b_4507_b5c2_8be7b15bb7d8.slice/crio-7a91d0c01053b510ddb47418753cd65f530cc75006f6404a1fa18bf14873b69c WatchSource:0}: Error finding container 7a91d0c01053b510ddb47418753cd65f530cc75006f6404a1fa18bf14873b69c: Status 404 returned error can't find the container with id 7a91d0c01053b510ddb47418753cd65f530cc75006f6404a1fa18bf14873b69c Jul 18 14:12:43.518642 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:43.518612 2564 generic.go:356] "Generic (PLEG): container finished" podID="b59354e9-f17b-4507-b5c2-8be7b15bb7d8" containerID="c81a24a5eebe93e3ed0210558704cbe5c151d142f66de8e60dc993f747ce2d3a" exitCode=0 Jul 18 14:12:43.519025 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:43.518714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" event={"ID":"b59354e9-f17b-4507-b5c2-8be7b15bb7d8","Type":"ContainerDied","Data":"c81a24a5eebe93e3ed0210558704cbe5c151d142f66de8e60dc993f747ce2d3a"} Jul 18 14:12:43.519025 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:43.518766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" event={"ID":"b59354e9-f17b-4507-b5c2-8be7b15bb7d8","Type":"ContainerStarted","Data":"7a91d0c01053b510ddb47418753cd65f530cc75006f6404a1fa18bf14873b69c"} Jul 18 14:12:44.658694 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:44.658669 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:44.762977 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:44.762946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59354e9-f17b-4507-b5c2-8be7b15bb7d8-maas-api-service-ca\") pod \"b59354e9-f17b-4507-b5c2-8be7b15bb7d8\" (UID: \"b59354e9-f17b-4507-b5c2-8be7b15bb7d8\") " Jul 18 14:12:44.763337 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:44.763313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59354e9-f17b-4507-b5c2-8be7b15bb7d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b59354e9-f17b-4507-b5c2-8be7b15bb7d8" (UID: "b59354e9-f17b-4507-b5c2-8be7b15bb7d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:12:44.864467 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:44.864358 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b59354e9-f17b-4507-b5c2-8be7b15bb7d8-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:12:45.531523 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:45.531493 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" Jul 18 14:12:45.531523 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:45.531498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn" event={"ID":"b59354e9-f17b-4507-b5c2-8be7b15bb7d8","Type":"ContainerDied","Data":"7a91d0c01053b510ddb47418753cd65f530cc75006f6404a1fa18bf14873b69c"} Jul 18 14:12:45.531740 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:12:45.531583 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a91d0c01053b510ddb47418753cd65f530cc75006f6404a1fa18bf14873b69c" Jul 18 14:13:02.398952 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:02.398860 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn"] Jul 18 14:13:02.401369 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:02.401344 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ee9e9-jx9cn"] Jul 18 14:13:02.681506 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:02.681423 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:13:03.973161 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:03.973130 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59354e9-f17b-4507-b5c2-8be7b15bb7d8" path="/var/lib/kubelet/pods/b59354e9-f17b-4507-b5c2-8be7b15bb7d8/volumes" Jul 18 14:13:08.583647 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:08.583606 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd"] Jul 18 14:13:08.584253 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:08.583832 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" containerID="cri-o://a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2" gracePeriod=30 Jul 18 14:13:09.815296 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:09.815258 2564 patch_prober.go:28] interesting pod/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 14:13:09.815706 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:09.815331 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 14:13:13.845852 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.845786 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:13:13.941529 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941498 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-token\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941739 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-socket\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941739 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-data\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941739 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941614 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-276fp\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-kube-api-access-276fp\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941739 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-certs\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941739 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-envoy\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941998 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-podinfo\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941998 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istiod-ca-cert\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941998 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941844 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-credential-socket\") pod \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\" (UID: \"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05\") " Jul 18 14:13:13.941998 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.941937 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:13:13.942207 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.942183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:13:13.942304 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.942269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:13:13.942362 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.942287 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:13.942362 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.942339 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:13.942605 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.942579 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:13:13.942667 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.942587 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:13:13.943969 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.943934 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:13:13.944070 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.943972 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-kube-api-access-276fp" (OuterVolumeSpecName: "kube-api-access-276fp") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "kube-api-access-276fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:13:13.944070 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.943978 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 14:13:13.944248 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:13.944234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" (UID: "3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:13:14.043420 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043356 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.043420 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043415 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276fp\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-kube-api-access-276fp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.043420 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043430 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.043667 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043442 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.043667 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043452 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.043667 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043462 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.043667 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.043473 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:13:14.700506 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.700468 2564 generic.go:356] "Generic (PLEG): container finished" podID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerID="a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2" exitCode=0 Jul 18 14:13:14.700696 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.700520 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" Jul 18 14:13:14.700696 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.700528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" event={"ID":"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05","Type":"ContainerDied","Data":"a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2"} Jul 18 14:13:14.700696 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.700573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd" event={"ID":"3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05","Type":"ContainerDied","Data":"06a84ace7d0cb0a973d660c80800620c8a25fbd406de801e9ded79fe8f5d4d96"} Jul 18 14:13:14.700696 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.700594 2564 scope.go:117] "RemoveContainer" containerID="a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2" Jul 18 14:13:14.710575 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.710273 2564 scope.go:117] "RemoveContainer" containerID="3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1" Jul 18 14:13:14.719127 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.719102 2564 scope.go:117] "RemoveContainer" containerID="a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2" Jul 18 14:13:14.719442 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:13:14.719423 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2\": container with ID starting with a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2 not found: ID does not exist" containerID="a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2" Jul 18 14:13:14.719525 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.719450 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2"} err="failed to get container status \"a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2\": rpc error: code = NotFound desc = could not find container \"a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2\": container with ID starting with a681f74bb81bd2ea6ccb7fcf035cac62118609252a3363666dc06b65d2441dc2 not found: ID does not exist" Jul 18 14:13:14.719525 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.719469 2564 scope.go:117] "RemoveContainer" containerID="3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1" Jul 18 14:13:14.719727 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:13:14.719705 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1\": container with ID starting with 3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1 not found: ID does not exist" containerID="3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1" Jul 18 14:13:14.719764 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.719732 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1"} err="failed to get container status \"3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1\": rpc error: code = NotFound desc = could not find container \"3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1\": container with ID starting with 3b206202f8b045885a66d781d7936e4f0f514ad3464d67781da5bc73dd424ce1 not found: ID does not exist" Jul 18 14:13:14.720487 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.720466 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd"] Jul 18 14:13:14.723936 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:14.723911 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6ee9e9-openshift-default-777bc54cf5-r29jd"] Jul 18 14:13:15.972457 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:15.972428 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" path="/var/lib/kubelet/pods/3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05/volumes" Jul 18 14:13:17.790849 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.790806 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm"] Jul 18 14:13:17.791364 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.791345 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" Jul 18 14:13:17.791446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.791366 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" Jul 18 14:13:17.791446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.791402 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59354e9-f17b-4507-b5c2-8be7b15bb7d8" containerName="revoke-keys" Jul 18 14:13:17.791446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.791410 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59354e9-f17b-4507-b5c2-8be7b15bb7d8" containerName="revoke-keys" Jul 18 14:13:17.791546 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.791493 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" Jul 18 14:13:17.791546 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.791503 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59354e9-f17b-4507-b5c2-8be7b15bb7d8" containerName="revoke-keys" Jul 18 14:13:17.796404 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.796367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.799047 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.799019 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-66d5d1-openshift-default-dockercfg-mvr9s\"" Jul 18 14:13:17.807936 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.807908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm"] Jul 18 14:13:17.877891 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.877858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-credential-socket\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.877908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpt4j\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-kube-api-access-vpt4j\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.877939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-podinfo\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.877961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-token\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.878044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-certs\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878077 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.878073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-envoy\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878241 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.878112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istiod-ca-cert\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878241 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.878170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-socket\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.878241 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.878193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-data\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979671 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-certs\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979671 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-envoy\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979927 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istiod-ca-cert\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979927 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-socket\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979927 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-data\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979927 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-credential-socket\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.979927 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpt4j\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-kube-api-access-vpt4j\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980188 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-podinfo\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980188 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.979966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-token\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980291 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.980223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-socket\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980435 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.980410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-credential-socket\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980513 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.980438 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istiod-ca-cert\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980558 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.980541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-certs\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.980594 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.980558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-data\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.982060 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.982043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-envoy\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.982397 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.982359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-podinfo\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.987900 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.987868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-token\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:17.988006 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:17.987994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpt4j\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-kube-api-access-vpt4j\") pod \"e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:18.107732 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.107637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:18.250714 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.250685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm"] Jul 18 14:13:18.251922 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:13:18.251888 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c74bb2e_a7b0_48d9_8c81_dfd3bbd737de.slice/crio-4d62971585115dd43d1bfe2a0aba557ca86ae2fba8c0748aa372018a16503fc0 WatchSource:0}: Error finding container 4d62971585115dd43d1bfe2a0aba557ca86ae2fba8c0748aa372018a16503fc0: Status 404 returned error can't find the container with id 4d62971585115dd43d1bfe2a0aba557ca86ae2fba8c0748aa372018a16503fc0 Jul 18 14:13:18.254074 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.254042 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:13:18.254152 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.254108 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:13:18.254152 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.254135 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:13:18.726134 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.726091 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" event={"ID":"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de","Type":"ContainerStarted","Data":"b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a"} Jul 18 14:13:18.726338 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.726194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" event={"ID":"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de","Type":"ContainerStarted","Data":"4d62971585115dd43d1bfe2a0aba557ca86ae2fba8c0748aa372018a16503fc0"} Jul 18 14:13:18.749241 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:18.749182 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podStartSLOduration=1.7491663229999999 podStartE2EDuration="1.749166323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:13:18.747610871 +0000 UTC m=+2449.297175577" watchObservedRunningTime="2026-07-18 14:13:18.749166323 +0000 UTC m=+2449.298731004" Jul 18 14:13:19.108758 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:19.108670 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:19.113991 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:19.113963 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:19.731524 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:19.731494 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:19.732436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:19.732409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:20.698327 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:20.698291 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:13:31.786464 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:31.786415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:13:35.022147 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.022103 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": read tcp 10.133.0.2:37130->10.133.0.66:15021: read: connection reset by peer" start-of-body= Jul 18 14:13:35.022563 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.022179 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": read tcp 10.133.0.2:37130->10.133.0.66:15021: read: connection reset by peer" Jul 18 14:13:35.825368 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.825335 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerID="b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a" exitCode=0 Jul 18 14:13:35.825552 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.825410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" event={"ID":"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de","Type":"ContainerDied","Data":"b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a"} Jul 18 14:13:35.825824 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.825810 2564 scope.go:117] "RemoveContainer" containerID="b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a" Jul 18 14:13:35.826833 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.826806 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:13:35.826894 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.826881 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:13:35.826929 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:35.826912 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 18 14:13:36.834507 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:36.834468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" event={"ID":"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de","Type":"ContainerStarted","Data":"a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5"} Jul 18 14:13:37.108367 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:37.108099 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:37.108367 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:37.108346 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 18 14:13:37.108638 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:37.108426 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 18 14:13:38.108925 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:38.108887 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 18 14:13:38.109338 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:38.108945 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 18 14:13:39.108695 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:39.108658 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 18 14:13:39.108905 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:39.108713 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 18 14:13:40.108609 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:40.108567 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 18 14:13:40.109041 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:40.108626 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 18 14:13:41.108815 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:41.108769 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 18 14:13:41.109209 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:41.108846 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 18 14:13:42.139983 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:42.139952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:42.140358 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:42.140243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:42.141052 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:42.141033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:13:53.186072 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:13:53.186036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:14:27.291846 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:14:27.291766 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:14:30.210509 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:14:30.210478 2564 scope.go:117] "RemoveContainer" containerID="ce69aa6f57309b89bc280095f87d5b0d488418aa8befce49b9e442f168de4652" Jul 18 14:14:44.090086 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:14:44.090048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:15:00.129895 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.129858 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m"] Jul 18 14:15:00.130538 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.130510 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" Jul 18 14:15:00.130538 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.130531 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" Jul 18 14:15:00.130664 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.130633 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d945bfb-61e1-47a7-a4ff-57aa7d2b5c05" containerName="istio-proxy" Jul 18 14:15:00.133860 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.133837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" Jul 18 14:15:00.135649 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.135627 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd"] Jul 18 14:15:00.136184 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.136166 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gkls\"" Jul 18 14:15:00.139248 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.139227 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm"] Jul 18 14:15:00.139402 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.139354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" Jul 18 14:15:00.142671 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.142649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m"] Jul 18 14:15:00.143817 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.142917 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" Jul 18 14:15:00.146730 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.146705 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd"] Jul 18 14:15:00.148509 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.148490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" Jul 18 14:15:00.150045 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.150021 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm"] Jul 18 14:15:00.158729 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.158708 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" Jul 18 14:15:00.187127 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.187104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" Jul 18 14:15:00.373857 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.373830 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm"] Jul 18 14:15:00.375645 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:15:00.375616 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc52f54_eb87_4718_a9b4_fceb155732fe.slice/crio-8d495275872953790cee6c2da3be04f9d6db1f6a9810505d146f24137b14e1a2 WatchSource:0}: Error finding container 8d495275872953790cee6c2da3be04f9d6db1f6a9810505d146f24137b14e1a2: Status 404 returned error can't find the container with id 8d495275872953790cee6c2da3be04f9d6db1f6a9810505d146f24137b14e1a2 Jul 18 14:15:00.521621 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.521589 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m"] Jul 18 14:15:00.523368 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:15:00.523342 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf71fa0_9cea_4a55_88cf_926b61d6d026.slice/crio-8f0e6f63a77cd093eda959daf46cea83cc419ac2144a5011bb2182dda3726fab WatchSource:0}: Error finding container 8f0e6f63a77cd093eda959daf46cea83cc419ac2144a5011bb2182dda3726fab: Status 404 returned error can't find the container with id 8f0e6f63a77cd093eda959daf46cea83cc419ac2144a5011bb2182dda3726fab Jul 18 14:15:00.555555 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:00.555530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd"] Jul 18 14:15:00.556964 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:15:00.556936 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386f8465_9042_43b3_be57_1bd56800c3cf.slice/crio-b759e56c4ecb9d3916df72f01d4e3c3b034e29da4f5351f2ed4e84a6805ee9d6 WatchSource:0}: Error finding container b759e56c4ecb9d3916df72f01d4e3c3b034e29da4f5351f2ed4e84a6805ee9d6: Status 404 returned error can't find the container with id b759e56c4ecb9d3916df72f01d4e3c3b034e29da4f5351f2ed4e84a6805ee9d6 Jul 18 14:15:01.328186 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.328148 2564 generic.go:356] "Generic (PLEG): container finished" podID="6bc52f54-eb87-4718-a9b4-fceb155732fe" containerID="c904d9967eacd96f9c5e9e48eb5e1733ace1fbc2d8473d4a65f7fe4eb20f6f85" exitCode=0 Jul 18 14:15:01.328615 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.328233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" event={"ID":"6bc52f54-eb87-4718-a9b4-fceb155732fe","Type":"ContainerDied","Data":"c904d9967eacd96f9c5e9e48eb5e1733ace1fbc2d8473d4a65f7fe4eb20f6f85"} Jul 18 14:15:01.328615 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.328291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" event={"ID":"6bc52f54-eb87-4718-a9b4-fceb155732fe","Type":"ContainerStarted","Data":"8d495275872953790cee6c2da3be04f9d6db1f6a9810505d146f24137b14e1a2"} Jul 18 14:15:01.329981 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.329952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" event={"ID":"386f8465-9042-43b3-be57-1bd56800c3cf","Type":"ContainerStarted","Data":"856246f72620cba880be0d0b30b813166c71a1d48ae3c1f1e194f2d3a3ed491e"} Jul 18 14:15:01.330121 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.330008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" event={"ID":"386f8465-9042-43b3-be57-1bd56800c3cf","Type":"ContainerStarted","Data":"b759e56c4ecb9d3916df72f01d4e3c3b034e29da4f5351f2ed4e84a6805ee9d6"} Jul 18 14:15:01.331337 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.331313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" event={"ID":"ccf71fa0-9cea-4a55-88cf-926b61d6d026","Type":"ContainerStarted","Data":"aa01cae00d428120986c17ab210d3680ed554d171dce279467c27a7e61d124b1"} Jul 18 14:15:01.331535 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.331366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" event={"ID":"ccf71fa0-9cea-4a55-88cf-926b61d6d026","Type":"ContainerStarted","Data":"8f0e6f63a77cd093eda959daf46cea83cc419ac2144a5011bb2182dda3726fab"} Jul 18 14:15:01.357319 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.357254 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" podStartSLOduration=1.357238812 podStartE2EDuration="1.357238812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:15:01.354715299 +0000 UTC m=+2551.904279992" watchObservedRunningTime="2026-07-18 14:15:01.357238812 +0000 UTC m=+2551.906803514" Jul 18 14:15:01.368424 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:01.368348 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" podStartSLOduration=1.368331316 podStartE2EDuration="1.368331316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:15:01.367144357 +0000 UTC m=+2551.916709040" watchObservedRunningTime="2026-07-18 14:15:01.368331316 +0000 UTC m=+2551.917895997" Jul 18 14:15:02.338433 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:02.338393 2564 generic.go:356] "Generic (PLEG): container finished" podID="386f8465-9042-43b3-be57-1bd56800c3cf" containerID="856246f72620cba880be0d0b30b813166c71a1d48ae3c1f1e194f2d3a3ed491e" exitCode=0 Jul 18 14:15:02.338871 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:02.338479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" event={"ID":"386f8465-9042-43b3-be57-1bd56800c3cf","Type":"ContainerDied","Data":"856246f72620cba880be0d0b30b813166c71a1d48ae3c1f1e194f2d3a3ed491e"} Jul 18 14:15:02.340110 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:02.340084 2564 generic.go:356] "Generic (PLEG): container finished" podID="ccf71fa0-9cea-4a55-88cf-926b61d6d026" containerID="aa01cae00d428120986c17ab210d3680ed554d171dce279467c27a7e61d124b1" exitCode=0 Jul 18 14:15:02.340211 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:02.340163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" event={"ID":"ccf71fa0-9cea-4a55-88cf-926b61d6d026","Type":"ContainerDied","Data":"aa01cae00d428120986c17ab210d3680ed554d171dce279467c27a7e61d124b1"} Jul 18 14:15:02.480863 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:02.480835 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" Jul 18 14:15:02.692310 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:02.692277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:15:03.348392 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:03.348349 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" Jul 18 14:15:03.348392 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:03.348345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm" event={"ID":"6bc52f54-eb87-4718-a9b4-fceb155732fe","Type":"ContainerDied","Data":"8d495275872953790cee6c2da3be04f9d6db1f6a9810505d146f24137b14e1a2"} Jul 18 14:15:03.348939 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:03.348418 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d495275872953790cee6c2da3be04f9d6db1f6a9810505d146f24137b14e1a2" Jul 18 14:15:03.485978 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:03.485952 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" Jul 18 14:15:03.516343 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:03.516315 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" Jul 18 14:15:04.356682 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:04.356600 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" Jul 18 14:15:04.357106 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:04.356607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739735-hs8rd" event={"ID":"386f8465-9042-43b3-be57-1bd56800c3cf","Type":"ContainerDied","Data":"b759e56c4ecb9d3916df72f01d4e3c3b034e29da4f5351f2ed4e84a6805ee9d6"} Jul 18 14:15:04.357106 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:04.356729 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b759e56c4ecb9d3916df72f01d4e3c3b034e29da4f5351f2ed4e84a6805ee9d6" Jul 18 14:15:04.357896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:04.357868 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" Jul 18 14:15:04.358024 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:04.357918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m" event={"ID":"ccf71fa0-9cea-4a55-88cf-926b61d6d026","Type":"ContainerDied","Data":"8f0e6f63a77cd093eda959daf46cea83cc419ac2144a5011bb2182dda3726fab"} Jul 18 14:15:04.358024 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:04.357961 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0e6f63a77cd093eda959daf46cea83cc419ac2144a5011bb2182dda3726fab" Jul 18 14:15:16.791652 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:16.791610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:15:34.089855 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:34.089814 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:15:52.585990 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:15:52.585888 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:16:11.480836 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:16:11.480800 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:16:28.986206 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:16:28.986162 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:16:39.188035 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:16:39.187994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:17:11.633396 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:17:11.633329 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:17:26.290501 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:17:26.290415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:17:30.251728 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:17:30.251691 2564 scope.go:117] "RemoveContainer" containerID="90e3813c20ffc97a6a0dddc0632f85e8604f21c78c99dd6017b44c5f837ad08d" Jul 18 14:17:48.284857 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:17:48.284806 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:18:02.578018 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:18:02.577980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:18:30.273368 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:18:30.273333 2564 scope.go:117] "RemoveContainer" containerID="e3be589b942f45b7236a7153b48ffce52e0859b97d5f71ba197c7be0a68a3fce" Jul 18 14:18:30.282470 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:18:30.282450 2564 scope.go:117] "RemoveContainer" containerID="4aa3d88a8146cf3d83ee48db2d830fe66111863f8d839b12046ee9944049cd45" Jul 18 14:18:33.884653 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:18:33.884611 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:18:52.182568 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:18:52.182465 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:19:15.080449 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:15.080407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:19:20.523937 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.523893 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-139623981489840"] Jul 18 14:19:20.524330 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524317 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf71fa0-9cea-4a55-88cf-926b61d6d026" containerName="cleanup" Jul 18 14:19:20.524330 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524331 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf71fa0-9cea-4a55-88cf-926b61d6d026" containerName="cleanup" Jul 18 14:19:20.524449 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524341 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc52f54-eb87-4718-a9b4-fceb155732fe" containerName="cleanup" Jul 18 14:19:20.524449 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524347 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc52f54-eb87-4718-a9b4-fceb155732fe" containerName="cleanup" Jul 18 14:19:20.524449 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524369 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386f8465-9042-43b3-be57-1bd56800c3cf" containerName="cleanup" Jul 18 14:19:20.524449 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524391 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="386f8465-9042-43b3-be57-1bd56800c3cf" containerName="cleanup" Jul 18 14:19:20.524576 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524452 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="386f8465-9042-43b3-be57-1bd56800c3cf" containerName="cleanup" Jul 18 14:19:20.524576 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524466 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc52f54-eb87-4718-a9b4-fceb155732fe" containerName="cleanup" Jul 18 14:19:20.524576 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.524474 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf71fa0-9cea-4a55-88cf-926b61d6d026" containerName="cleanup" Jul 18 14:19:20.527696 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.527678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:20.530976 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.530949 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8t7st\"" Jul 18 14:19:20.533589 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.533559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623981489840"] Jul 18 14:19:20.709247 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.709213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42thd\" (UniqueName: \"kubernetes.io/projected/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93-kube-api-access-42thd\") pod \"test-curl-7648-139623981489840\" (UID: \"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93\") " pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:20.810116 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.810027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42thd\" (UniqueName: \"kubernetes.io/projected/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93-kube-api-access-42thd\") pod \"test-curl-7648-139623981489840\" (UID: \"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93\") " pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:20.818299 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.818261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42thd\" (UniqueName: \"kubernetes.io/projected/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93-kube-api-access-42thd\") pod \"test-curl-7648-139623981489840\" (UID: \"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93\") " pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:20.839181 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.839148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:20.981607 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.981575 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623981489840"] Jul 18 14:19:20.982815 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:19:20.982790 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d8beee0_69e8_4c30_baaf_cb50fc2d3d93.slice/crio-e0bb462e289916840ecb5a0b5218fecb48cbf55eb5137e7fa10f117016dff080 WatchSource:0}: Error finding container e0bb462e289916840ecb5a0b5218fecb48cbf55eb5137e7fa10f117016dff080: Status 404 returned error can't find the container with id e0bb462e289916840ecb5a0b5218fecb48cbf55eb5137e7fa10f117016dff080 Jul 18 14:19:20.984224 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:20.984204 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 14:19:21.862825 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:21.862794 2564 generic.go:356] "Generic (PLEG): container finished" podID="3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" containerID="9c8375823c7e01143711f584a0f461cfdaca4324214b701d8c70c8c03b2adf76" exitCode=0 Jul 18 14:19:21.863208 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:21.862889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623981489840" event={"ID":"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93","Type":"ContainerDied","Data":"9c8375823c7e01143711f584a0f461cfdaca4324214b701d8c70c8c03b2adf76"} Jul 18 14:19:21.863208 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:21.862933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623981489840" event={"ID":"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93","Type":"ContainerStarted","Data":"e0bb462e289916840ecb5a0b5218fecb48cbf55eb5137e7fa10f117016dff080"} Jul 18 14:19:23.005842 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.005814 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:23.028027 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.027996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42thd\" (UniqueName: \"kubernetes.io/projected/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93-kube-api-access-42thd\") pod \"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93\" (UID: \"3d8beee0-69e8-4c30-baaf-cb50fc2d3d93\") " Jul 18 14:19:23.030133 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.030101 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93-kube-api-access-42thd" (OuterVolumeSpecName: "kube-api-access-42thd") pod "3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" (UID: "3d8beee0-69e8-4c30-baaf-cb50fc2d3d93"). InnerVolumeSpecName "kube-api-access-42thd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:19:23.040164 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.040135 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7648-139623981489840_3d8beee0-69e8-4c30-baaf-cb50fc2d3d93/test-curl-7648-139623981489840/0.log" Jul 18 14:19:23.061575 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.061538 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-139623981489840"] Jul 18 14:19:23.063852 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.063830 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-139623981489840"] Jul 18 14:19:23.129607 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.129514 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42thd\" (UniqueName: \"kubernetes.io/projected/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93-kube-api-access-42thd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:19:23.178522 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.178488 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-139623981487280"] Jul 18 14:19:23.178932 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.178917 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" containerName="test-curl-7648-139623981489840" Jul 18 14:19:23.178975 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.178934 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" containerName="test-curl-7648-139623981489840" Jul 18 14:19:23.179016 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.179008 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" containerName="test-curl-7648-139623981489840" Jul 18 14:19:23.182025 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.182010 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:23.187837 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.187807 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623981487280"] Jul 18 14:19:23.230920 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.230884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqnn\" (UniqueName: \"kubernetes.io/projected/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253-kube-api-access-7wqnn\") pod \"test-curl-7648-139623981487280\" (UID: \"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253\") " pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:23.332097 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.332058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqnn\" (UniqueName: \"kubernetes.io/projected/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253-kube-api-access-7wqnn\") pod \"test-curl-7648-139623981487280\" (UID: \"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253\") " pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:23.341141 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.341113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqnn\" (UniqueName: \"kubernetes.io/projected/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253-kube-api-access-7wqnn\") pod \"test-curl-7648-139623981487280\" (UID: \"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253\") " pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:23.493132 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.493099 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:23.635122 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.635092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623981487280"] Jul 18 14:19:23.636063 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:19:23.636039 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4a1fc41_ff0a_43cc_9b85_25fedd5fb253.slice/crio-bf9b58ac33da09725c8b24c5618eb72137a1d2e82a69cc7a125f63a3f496a291 WatchSource:0}: Error finding container bf9b58ac33da09725c8b24c5618eb72137a1d2e82a69cc7a125f63a3f496a291: Status 404 returned error can't find the container with id bf9b58ac33da09725c8b24c5618eb72137a1d2e82a69cc7a125f63a3f496a291 Jul 18 14:19:23.875084 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.874991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623981487280" event={"ID":"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253","Type":"ContainerStarted","Data":"bf9b58ac33da09725c8b24c5618eb72137a1d2e82a69cc7a125f63a3f496a291"} Jul 18 14:19:23.876267 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.876244 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:23.876267 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.876265 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0bb462e289916840ecb5a0b5218fecb48cbf55eb5137e7fa10f117016dff080" Jul 18 14:19:23.887193 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:23.887159 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623981489840\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" pod="opendatahub/test-curl-7648-139623981489840" Jul 18 14:19:23.974311 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:23.974284 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8beee0-69e8-4c30-baaf-cb50fc2d3d93" path="/var/lib/kubelet/pods/3d8beee0-69e8-4c30-baaf-cb50fc2d3d93/volumes" Jul 18 14:19:24.883967 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:24.883930 2564 generic.go:356] "Generic (PLEG): container finished" podID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" containerID="9b71421b6d2ad01d0a64158f50f6fc64ae189d9c374792e87620bc7e3ffe840e" exitCode=0 Jul 18 14:19:24.884389 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:24.884012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623981487280" event={"ID":"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253","Type":"ContainerDied","Data":"9b71421b6d2ad01d0a64158f50f6fc64ae189d9c374792e87620bc7e3ffe840e"} Jul 18 14:19:26.027085 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.027058 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:26.045123 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.045083 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7648-139623981487280_f4a1fc41-ff0a-43cc-9b85-25fedd5fb253/test-curl-7648-139623981487280/0.log" Jul 18 14:19:26.056454 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.056427 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wqnn\" (UniqueName: \"kubernetes.io/projected/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253-kube-api-access-7wqnn\") pod \"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253\" (UID: \"f4a1fc41-ff0a-43cc-9b85-25fedd5fb253\") " Jul 18 14:19:26.058842 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.058813 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253-kube-api-access-7wqnn" (OuterVolumeSpecName: "kube-api-access-7wqnn") pod "f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" (UID: "f4a1fc41-ff0a-43cc-9b85-25fedd5fb253"). InnerVolumeSpecName "kube-api-access-7wqnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:19:26.066701 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.066674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-139623981487280"] Jul 18 14:19:26.070703 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.070680 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-139623981487280"] Jul 18 14:19:26.157304 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.157212 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqnn\" (UniqueName: \"kubernetes.io/projected/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253-kube-api-access-7wqnn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:19:26.223531 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.223495 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:26.223917 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.223904 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" containerName="test-curl-7648-139623981487280" Jul 18 14:19:26.223960 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.223919 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" containerName="test-curl-7648-139623981487280" Jul 18 14:19:26.223992 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.223982 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" containerName="test-curl-7648-139623981487280" Jul 18 14:19:26.226986 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.226970 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:26.232557 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.232530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:26.257820 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.257785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcv5\" (UniqueName: \"kubernetes.io/projected/b65216db-8139-4e31-9516-e2a193935072-kube-api-access-5xcv5\") pod \"test-curl-7648-139623985844848\" (UID: \"b65216db-8139-4e31-9516-e2a193935072\") " pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:26.359190 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.359148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcv5\" (UniqueName: \"kubernetes.io/projected/b65216db-8139-4e31-9516-e2a193935072-kube-api-access-5xcv5\") pod \"test-curl-7648-139623985844848\" (UID: \"b65216db-8139-4e31-9516-e2a193935072\") " pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:26.367165 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.367137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcv5\" (UniqueName: \"kubernetes.io/projected/b65216db-8139-4e31-9516-e2a193935072-kube-api-access-5xcv5\") pod \"test-curl-7648-139623985844848\" (UID: \"b65216db-8139-4e31-9516-e2a193935072\") " pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:26.537985 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.537948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:26.672308 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.672278 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:26.673525 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:19:26.673501 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb65216db_8139_4e31_9516_e2a193935072.slice/crio-fbf7ebd03a64b6a5e21bbb1e66ffc3bfc6ee3546557778bf5ed0ce35d5e1791c WatchSource:0}: Error finding container fbf7ebd03a64b6a5e21bbb1e66ffc3bfc6ee3546557778bf5ed0ce35d5e1791c: Status 404 returned error can't find the container with id fbf7ebd03a64b6a5e21bbb1e66ffc3bfc6ee3546557778bf5ed0ce35d5e1791c Jul 18 14:19:26.897700 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.897608 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:26.897700 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.897618 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9b58ac33da09725c8b24c5618eb72137a1d2e82a69cc7a125f63a3f496a291" Jul 18 14:19:26.898797 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:26.898769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623985844848" event={"ID":"b65216db-8139-4e31-9516-e2a193935072","Type":"ContainerStarted","Data":"fbf7ebd03a64b6a5e21bbb1e66ffc3bfc6ee3546557778bf5ed0ce35d5e1791c"} Jul 18 14:19:26.908481 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:26.908452 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623981487280\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:27.906618 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:27.906579 2564 generic.go:356] "Generic (PLEG): container finished" podID="b65216db-8139-4e31-9516-e2a193935072" containerID="5a3876dc1a6bae2f0f01eb610dc058be69f16ff1ac47ed50f0fd699bd809ae21" exitCode=0 Jul 18 14:19:27.906997 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:27.906638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623985844848" event={"ID":"b65216db-8139-4e31-9516-e2a193935072","Type":"ContainerDied","Data":"5a3876dc1a6bae2f0f01eb610dc058be69f16ff1ac47ed50f0fd699bd809ae21"} Jul 18 14:19:27.908913 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:27.908890 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623981487280\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" pod="opendatahub/test-curl-7648-139623981487280" Jul 18 14:19:27.973297 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:27.973257 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a1fc41-ff0a-43cc-9b85-25fedd5fb253" path="/var/lib/kubelet/pods/f4a1fc41-ff0a-43cc-9b85-25fedd5fb253/volumes" Jul 18 14:19:29.050080 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.050050 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.069587 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.069548 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7648-139623985844848_b65216db-8139-4e31-9516-e2a193935072/test-curl-7648-139623985844848/0.log" Jul 18 14:19:29.085783 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.085743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xcv5\" (UniqueName: \"kubernetes.io/projected/b65216db-8139-4e31-9516-e2a193935072-kube-api-access-5xcv5\") pod \"b65216db-8139-4e31-9516-e2a193935072\" (UID: \"b65216db-8139-4e31-9516-e2a193935072\") " Jul 18 14:19:29.087906 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.087867 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65216db-8139-4e31-9516-e2a193935072-kube-api-access-5xcv5" (OuterVolumeSpecName: "kube-api-access-5xcv5") pod "b65216db-8139-4e31-9516-e2a193935072" (UID: "b65216db-8139-4e31-9516-e2a193935072"). InnerVolumeSpecName "kube-api-access-5xcv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:19:29.093130 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.093092 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:29.096981 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.096957 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:29.186572 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.186545 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcv5\" (UniqueName: \"kubernetes.io/projected/b65216db-8139-4e31-9516-e2a193935072-kube-api-access-5xcv5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:19:29.210394 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.210338 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:29.210790 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.210774 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65216db-8139-4e31-9516-e2a193935072" containerName="test-curl-7648-139623985844848" Jul 18 14:19:29.210832 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.210792 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65216db-8139-4e31-9516-e2a193935072" containerName="test-curl-7648-139623985844848" Jul 18 14:19:29.210886 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.210877 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65216db-8139-4e31-9516-e2a193935072" containerName="test-curl-7648-139623985844848" Jul 18 14:19:29.213982 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.213965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.220800 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.220740 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:29.287003 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.286967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4kz\" (UniqueName: \"kubernetes.io/projected/4ea6c558-c926-4f8a-8b5b-977ca4907e8d-kube-api-access-5d4kz\") pod \"test-curl-7648-139623985844848\" (UID: \"4ea6c558-c926-4f8a-8b5b-977ca4907e8d\") " pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.387754 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.387713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4kz\" (UniqueName: \"kubernetes.io/projected/4ea6c558-c926-4f8a-8b5b-977ca4907e8d-kube-api-access-5d4kz\") pod \"test-curl-7648-139623985844848\" (UID: \"4ea6c558-c926-4f8a-8b5b-977ca4907e8d\") " pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.396098 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.396068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4kz\" (UniqueName: \"kubernetes.io/projected/4ea6c558-c926-4f8a-8b5b-977ca4907e8d-kube-api-access-5d4kz\") pod \"test-curl-7648-139623985844848\" (UID: \"4ea6c558-c926-4f8a-8b5b-977ca4907e8d\") " pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.524525 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.524431 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.665131 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.665104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:29.667363 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:19:29.667337 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea6c558_c926_4f8a_8b5b_977ca4907e8d.slice/crio-37b82b8e6cc058df70b378879b410e6964e2d37ace39b0cbcfec90c6d8e0a2be WatchSource:0}: Error finding container 37b82b8e6cc058df70b378879b410e6964e2d37ace39b0cbcfec90c6d8e0a2be: Status 404 returned error can't find the container with id 37b82b8e6cc058df70b378879b410e6964e2d37ace39b0cbcfec90c6d8e0a2be Jul 18 14:19:29.919310 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.919214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623985844848" event={"ID":"4ea6c558-c926-4f8a-8b5b-977ca4907e8d","Type":"ContainerStarted","Data":"37b82b8e6cc058df70b378879b410e6964e2d37ace39b0cbcfec90c6d8e0a2be"} Jul 18 14:19:29.920593 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.920573 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:29.920593 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.920586 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf7ebd03a64b6a5e21bbb1e66ffc3bfc6ee3546557778bf5ed0ce35d5e1791c" Jul 18 14:19:29.932623 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.932587 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7648-139623985844848" oldPodUID="b65216db-8139-4e31-9516-e2a193935072" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" Jul 18 14:19:29.974350 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:29.974319 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65216db-8139-4e31-9516-e2a193935072" path="/var/lib/kubelet/pods/b65216db-8139-4e31-9516-e2a193935072/volumes" Jul 18 14:19:30.306951 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:30.306918 2564 scope.go:117] "RemoveContainer" containerID="c81a24a5eebe93e3ed0210558704cbe5c151d142f66de8e60dc993f747ce2d3a" Jul 18 14:19:30.929160 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:30.929123 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" containerID="5d3de7b297f4ae0ba233b9bea96848661cd131a9b9c6ea3dcfa153f1e213b9a1" exitCode=0 Jul 18 14:19:30.929330 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:30.929208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623985844848" event={"ID":"4ea6c558-c926-4f8a-8b5b-977ca4907e8d","Type":"ContainerDied","Data":"5d3de7b297f4ae0ba233b9bea96848661cd131a9b9c6ea3dcfa153f1e213b9a1"} Jul 18 14:19:31.086705 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:31.086659 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:19:32.076400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.076351 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:32.095008 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.094979 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7648-139623985844848_4ea6c558-c926-4f8a-8b5b-977ca4907e8d/test-curl-7648-139623985844848/0.log" Jul 18 14:19:32.110819 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.110789 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d4kz\" (UniqueName: \"kubernetes.io/projected/4ea6c558-c926-4f8a-8b5b-977ca4907e8d-kube-api-access-5d4kz\") pod \"4ea6c558-c926-4f8a-8b5b-977ca4907e8d\" (UID: \"4ea6c558-c926-4f8a-8b5b-977ca4907e8d\") " Jul 18 14:19:32.113040 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.112995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea6c558-c926-4f8a-8b5b-977ca4907e8d-kube-api-access-5d4kz" (OuterVolumeSpecName: "kube-api-access-5d4kz") pod "4ea6c558-c926-4f8a-8b5b-977ca4907e8d" (UID: "4ea6c558-c926-4f8a-8b5b-977ca4907e8d"). InnerVolumeSpecName "kube-api-access-5d4kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:19:32.119549 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.119425 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:32.124674 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.124632 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-139623985844848"] Jul 18 14:19:32.212203 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.212107 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4kz\" (UniqueName: \"kubernetes.io/projected/4ea6c558-c926-4f8a-8b5b-977ca4907e8d-kube-api-access-5d4kz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:19:32.234173 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.234130 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:32.234601 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.234584 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" containerName="test-curl-7648-139623985844848" Jul 18 14:19:32.234601 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.234602 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" containerName="test-curl-7648-139623985844848" Jul 18 14:19:32.234702 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.234675 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" containerName="test-curl-7648-139623985844848" Jul 18 14:19:32.237776 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.237760 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:32.243461 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.243430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:32.313250 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.313214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdhk7\" (UniqueName: \"kubernetes.io/projected/61ec7e57-5c44-4d15-8a65-2e710f65876f-kube-api-access-qdhk7\") pod \"test-curl-7648-139623983511856\" (UID: \"61ec7e57-5c44-4d15-8a65-2e710f65876f\") " pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:32.413749 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.413706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdhk7\" (UniqueName: \"kubernetes.io/projected/61ec7e57-5c44-4d15-8a65-2e710f65876f-kube-api-access-qdhk7\") pod \"test-curl-7648-139623983511856\" (UID: \"61ec7e57-5c44-4d15-8a65-2e710f65876f\") " pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:32.421617 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.421593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdhk7\" (UniqueName: \"kubernetes.io/projected/61ec7e57-5c44-4d15-8a65-2e710f65876f-kube-api-access-qdhk7\") pod \"test-curl-7648-139623983511856\" (UID: \"61ec7e57-5c44-4d15-8a65-2e710f65876f\") " pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:32.549306 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.549215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:32.681530 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.681504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:32.682628 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:19:32.682604 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61ec7e57_5c44_4d15_8a65_2e710f65876f.slice/crio-c246863d3ca8da3b68f3672454dbfe8d2d552075bb7d8cb89c9615182ecfb678 WatchSource:0}: Error finding container c246863d3ca8da3b68f3672454dbfe8d2d552075bb7d8cb89c9615182ecfb678: Status 404 returned error can't find the container with id c246863d3ca8da3b68f3672454dbfe8d2d552075bb7d8cb89c9615182ecfb678 Jul 18 14:19:32.942362 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.942327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623983511856" event={"ID":"61ec7e57-5c44-4d15-8a65-2e710f65876f","Type":"ContainerStarted","Data":"c246863d3ca8da3b68f3672454dbfe8d2d552075bb7d8cb89c9615182ecfb678"} Jul 18 14:19:32.943593 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.943572 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37b82b8e6cc058df70b378879b410e6964e2d37ace39b0cbcfec90c6d8e0a2be" Jul 18 14:19:32.943709 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:32.943608 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:32.954012 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:32.953981 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623985844848\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:33.950608 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:33.950573 2564 generic.go:356] "Generic (PLEG): container finished" podID="61ec7e57-5c44-4d15-8a65-2e710f65876f" containerID="f03b3b3bcfa5e28faa8b0fc55ec2dd4686f904010f772891900b42c9d03b63f5" exitCode=0 Jul 18 14:19:33.950970 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:33.950665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623983511856" event={"ID":"61ec7e57-5c44-4d15-8a65-2e710f65876f","Type":"ContainerDied","Data":"f03b3b3bcfa5e28faa8b0fc55ec2dd4686f904010f772891900b42c9d03b63f5"} Jul 18 14:19:33.965575 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:33.965534 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623985844848\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" pod="opendatahub/test-curl-7648-139623985844848" Jul 18 14:19:33.972978 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:33.972950 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea6c558-c926-4f8a-8b5b-977ca4907e8d" path="/var/lib/kubelet/pods/4ea6c558-c926-4f8a-8b5b-977ca4907e8d/volumes" Jul 18 14:19:35.095630 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.095604 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.117873 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.117844 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7648-139623983511856_61ec7e57-5c44-4d15-8a65-2e710f65876f/test-curl-7648-139623983511856/0.log" Jul 18 14:19:35.139503 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.139470 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdhk7\" (UniqueName: \"kubernetes.io/projected/61ec7e57-5c44-4d15-8a65-2e710f65876f-kube-api-access-qdhk7\") pod \"61ec7e57-5c44-4d15-8a65-2e710f65876f\" (UID: \"61ec7e57-5c44-4d15-8a65-2e710f65876f\") " Jul 18 14:19:35.140204 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.140178 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:35.141844 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.141812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ec7e57-5c44-4d15-8a65-2e710f65876f-kube-api-access-qdhk7" (OuterVolumeSpecName: "kube-api-access-qdhk7") pod "61ec7e57-5c44-4d15-8a65-2e710f65876f" (UID: "61ec7e57-5c44-4d15-8a65-2e710f65876f"). InnerVolumeSpecName "kube-api-access-qdhk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:19:35.142617 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.142595 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:35.240570 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.240487 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdhk7\" (UniqueName: \"kubernetes.io/projected/61ec7e57-5c44-4d15-8a65-2e710f65876f-kube-api-access-qdhk7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:19:35.255697 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.255660 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:35.256122 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.256107 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ec7e57-5c44-4d15-8a65-2e710f65876f" containerName="test-curl-7648-139623983511856" Jul 18 14:19:35.256184 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.256123 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ec7e57-5c44-4d15-8a65-2e710f65876f" containerName="test-curl-7648-139623983511856" Jul 18 14:19:35.256222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.256196 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ec7e57-5c44-4d15-8a65-2e710f65876f" containerName="test-curl-7648-139623983511856" Jul 18 14:19:35.259369 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.259351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.263737 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.263711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:35.341565 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.341526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pll2\" (UniqueName: \"kubernetes.io/projected/dd9851c4-ba33-4e13-8191-95d3cbc13ae8-kube-api-access-7pll2\") pod \"test-curl-7648-139623983511856\" (UID: \"dd9851c4-ba33-4e13-8191-95d3cbc13ae8\") " pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.442105 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.442067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pll2\" (UniqueName: \"kubernetes.io/projected/dd9851c4-ba33-4e13-8191-95d3cbc13ae8-kube-api-access-7pll2\") pod \"test-curl-7648-139623983511856\" (UID: \"dd9851c4-ba33-4e13-8191-95d3cbc13ae8\") " pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.450213 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.450181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pll2\" (UniqueName: \"kubernetes.io/projected/dd9851c4-ba33-4e13-8191-95d3cbc13ae8-kube-api-access-7pll2\") pod \"test-curl-7648-139623983511856\" (UID: \"dd9851c4-ba33-4e13-8191-95d3cbc13ae8\") " pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.570154 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.570065 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.708227 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.708199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:35.709390 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:19:35.709350 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd9851c4_ba33_4e13_8191_95d3cbc13ae8.slice/crio-72bf91a74279f568f4824058bb61ac115a152b21d0a7d46d50bb7bb95c8a83cb WatchSource:0}: Error finding container 72bf91a74279f568f4824058bb61ac115a152b21d0a7d46d50bb7bb95c8a83cb: Status 404 returned error can't find the container with id 72bf91a74279f568f4824058bb61ac115a152b21d0a7d46d50bb7bb95c8a83cb Jul 18 14:19:35.965534 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.965498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623983511856" event={"ID":"dd9851c4-ba33-4e13-8191-95d3cbc13ae8","Type":"ContainerStarted","Data":"72bf91a74279f568f4824058bb61ac115a152b21d0a7d46d50bb7bb95c8a83cb"} Jul 18 14:19:35.966799 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.966773 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:35.974542 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.974511 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ec7e57-5c44-4d15-8a65-2e710f65876f" path="/var/lib/kubelet/pods/61ec7e57-5c44-4d15-8a65-2e710f65876f/volumes" Jul 18 14:19:35.975007 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:35.974993 2564 scope.go:117] "RemoveContainer" containerID="f03b3b3bcfa5e28faa8b0fc55ec2dd4686f904010f772891900b42c9d03b63f5" Jul 18 14:19:36.974205 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:36.974167 2564 generic.go:356] "Generic (PLEG): container finished" podID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" containerID="ce210a0301a4470ecc40c67f366a007176a060428b65da79aac78e5b736f44b8" exitCode=0 Jul 18 14:19:36.974710 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:36.974256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7648-139623983511856" event={"ID":"dd9851c4-ba33-4e13-8191-95d3cbc13ae8","Type":"ContainerDied","Data":"ce210a0301a4470ecc40c67f366a007176a060428b65da79aac78e5b736f44b8"} Jul 18 14:19:38.128434 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.128407 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:38.148659 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.148625 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7648-139623983511856_dd9851c4-ba33-4e13-8191-95d3cbc13ae8/test-curl-7648-139623983511856/0.log" Jul 18 14:19:38.170748 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.170712 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pll2\" (UniqueName: \"kubernetes.io/projected/dd9851c4-ba33-4e13-8191-95d3cbc13ae8-kube-api-access-7pll2\") pod \"dd9851c4-ba33-4e13-8191-95d3cbc13ae8\" (UID: \"dd9851c4-ba33-4e13-8191-95d3cbc13ae8\") " Jul 18 14:19:38.172745 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.172710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9851c4-ba33-4e13-8191-95d3cbc13ae8-kube-api-access-7pll2" (OuterVolumeSpecName: "kube-api-access-7pll2") pod "dd9851c4-ba33-4e13-8191-95d3cbc13ae8" (UID: "dd9851c4-ba33-4e13-8191-95d3cbc13ae8"). InnerVolumeSpecName "kube-api-access-7pll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:19:38.177087 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.177062 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:38.179598 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.179579 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7648-139623983511856"] Jul 18 14:19:38.272584 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.272494 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pll2\" (UniqueName: \"kubernetes.io/projected/dd9851c4-ba33-4e13-8191-95d3cbc13ae8-kube-api-access-7pll2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:19:38.989386 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.989359 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72bf91a74279f568f4824058bb61ac115a152b21d0a7d46d50bb7bb95c8a83cb" Jul 18 14:19:38.989386 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:38.989389 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:38.999257 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:38.999227 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623983511856\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:39.972913 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:19:39.972875 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7648-139623983511856\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" pod="opendatahub/test-curl-7648-139623983511856" Jul 18 14:19:39.973941 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:39.973919 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" path="/var/lib/kubelet/pods/dd9851c4-ba33-4e13-8191-95d3cbc13ae8/volumes" Jul 18 14:19:48.731589 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:48.731555 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m"] Jul 18 14:19:48.737210 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:48.737177 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1bdd6b-29739735-dtt9m"] Jul 18 14:19:49.973175 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:49.973139 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccf71fa0-9cea-4a55-88cf-926b61d6d026" path="/var/lib/kubelet/pods/ccf71fa0-9cea-4a55-88cf-926b61d6d026/volumes" Jul 18 14:19:51.680249 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:19:51.680215 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:20:10.947973 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:10.947937 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm"] Jul 18 14:20:10.950156 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:10.950132 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-66d5d1-29739735-hnhzm"] Jul 18 14:20:11.973172 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:11.973137 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc52f54-eb87-4718-a9b4-fceb155732fe" path="/var/lib/kubelet/pods/6bc52f54-eb87-4718-a9b4-fceb155732fe/volumes" Jul 18 14:20:13.384160 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:13.384125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tw58w"] Jul 18 14:20:21.936844 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:21.936748 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm"] Jul 18 14:20:21.937449 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:21.936999 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" containerID="cri-o://a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5" gracePeriod=30 Jul 18 14:20:27.208475 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.208410 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:20:27.279277 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.279239 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerID="a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5" exitCode=0 Jul 18 14:20:27.279481 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.279294 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" Jul 18 14:20:27.279481 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.279323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" event={"ID":"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de","Type":"ContainerDied","Data":"a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5"} Jul 18 14:20:27.279481 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.279407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" event={"ID":"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de","Type":"ContainerDied","Data":"4d62971585115dd43d1bfe2a0aba557ca86ae2fba8c0748aa372018a16503fc0"} Jul 18 14:20:27.279481 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.279434 2564 scope.go:117] "RemoveContainer" containerID="a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5" Jul 18 14:20:27.289149 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.289127 2564 scope.go:117] "RemoveContainer" containerID="b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a" Jul 18 14:20:27.297887 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.297863 2564 scope.go:117] "RemoveContainer" containerID="a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5" Jul 18 14:20:27.298153 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:20:27.298135 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5\": container with ID starting with a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5 not found: ID does not exist" containerID="a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5" Jul 18 14:20:27.298194 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.298163 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5"} err="failed to get container status \"a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5\": rpc error: code = NotFound desc = could not find container \"a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5\": container with ID starting with a87c3a270e175cdf03b0487ecdafc831e6f8af2c329a152df618e2645ad635a5 not found: ID does not exist" Jul 18 14:20:27.298194 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.298183 2564 scope.go:117] "RemoveContainer" containerID="b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a" Jul 18 14:20:27.298449 ip-10-0-136-156 kubenswrapper[2564]: E0718 14:20:27.298429 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a\": container with ID starting with b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a not found: ID does not exist" containerID="b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a" Jul 18 14:20:27.298522 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.298455 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a"} err="failed to get container status \"b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a\": rpc error: code = NotFound desc = could not find container \"b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a\": container with ID starting with b37114676289557d32a39410b982b9cfc428e7fb861566923a9e295bf5a1107a not found: ID does not exist" Jul 18 14:20:27.336446 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336420 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-socket\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336627 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336454 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-token\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336627 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336490 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpt4j\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-kube-api-access-vpt4j\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336627 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-certs\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336627 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336530 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istiod-ca-cert\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336627 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336584 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-envoy\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336664 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-credential-socket\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336702 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-podinfo\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336727 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-data\") pod \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\" (UID: \"8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de\") " Jul 18 14:20:27.336896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336771 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:20:27.337119 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.336955 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:20:27.337119 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.337025 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 14:20:27.337243 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.337133 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:20:27.337243 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.337217 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.337243 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.337237 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.337434 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.337250 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.337742 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.337721 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:20:27.338913 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.338828 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 14:20:27.339033 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.338962 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-kube-api-access-vpt4j" (OuterVolumeSpecName: "kube-api-access-vpt4j") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "kube-api-access-vpt4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:20:27.339102 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.339038 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 14:20:27.339289 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.339265 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" (UID: "8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 14:20:27.438616 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.438572 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.438616 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.438606 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.438616 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.438616 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.438616 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.438624 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.438896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.438633 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpt4j\" (UniqueName: \"kubernetes.io/projected/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-kube-api-access-vpt4j\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.438896 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.438641 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 18 14:20:27.603119 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.603079 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm"] Jul 18 14:20:27.606433 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.606405 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm"] Jul 18 14:20:27.627460 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.627429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pfrd7_1d32444b-58be-4066-b0d0-bd3451756721/manager/0.log" Jul 18 14:20:27.964460 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.964430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w8txs_fac34956-9721-4e24-9f84-87fde2c723f7/server/0.log" Jul 18 14:20:27.973391 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:27.973346 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" path="/var/lib/kubelet/pods/8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de/volumes" Jul 18 14:20:28.078461 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.078425 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mcpjl_0492ceb8-72fc-43ea-b95b-45c657ff946d/manager/0.log" Jul 18 14:20:28.141208 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.141170 2564 patch_prober.go:28] interesting pod/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 18 14:20:28.141389 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.141248 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-66d5d1-openshift-default-5648967b55-zr6rm" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 18 14:20:28.440481 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.440454 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7687b4d-shppk_d179a8f6-f972-46cc-83b1-409934056e12/copy-manifests/0.log" Jul 18 14:20:28.446358 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.446336 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7687b4d-shppk_d179a8f6-f972-46cc-83b1-409934056e12/manager/0.log" Jul 18 14:20:28.770094 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.770061 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-689d469879-6d7n7_0097e143-75f2-4a20-bbcc-09c29df116c3/maas-api/0.log" Jul 18 14:20:28.877406 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.877347 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29739720-xbzvz_53532842-c490-4dad-9c2e-eafb3eddfc73/cleanup/0.log" Jul 18 14:20:28.982368 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:28.982340 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29739735-hs8rd_386f8465-9042-43b3-be57-1bd56800c3cf/cleanup/0.log" Jul 18 14:20:29.827368 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:29.827336 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9_b345f19e-eadb-4cfa-b1f4-e76c18f3257a/pull/0.log" Jul 18 14:20:29.840207 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:29.840177 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9_b345f19e-eadb-4cfa-b1f4-e76c18f3257a/extract/0.log" Jul 18 14:20:29.845585 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:29.845553 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9_b345f19e-eadb-4cfa-b1f4-e76c18f3257a/util/0.log" Jul 18 14:20:29.947805 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:29.947768 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c_79eff395-f999-40bc-96cb-164545194b03/util/0.log" Jul 18 14:20:29.953322 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:29.953218 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c_79eff395-f999-40bc-96cb-164545194b03/pull/0.log" Jul 18 14:20:29.958438 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:29.958422 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c_79eff395-f999-40bc-96cb-164545194b03/extract/0.log" Jul 18 14:20:30.067222 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.067192 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29_72b7aeca-8a94-4994-8f83-1f2061afe4c1/util/0.log" Jul 18 14:20:30.072557 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.072528 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29_72b7aeca-8a94-4994-8f83-1f2061afe4c1/pull/0.log" Jul 18 14:20:30.077719 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.077658 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29_72b7aeca-8a94-4994-8f83-1f2061afe4c1/extract/0.log" Jul 18 14:20:30.183390 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.183352 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz_b5470b4e-48bc-4c6e-a8a2-ada8d7636789/pull/0.log" Jul 18 14:20:30.188679 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.188651 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz_b5470b4e-48bc-4c6e-a8a2-ada8d7636789/extract/0.log" Jul 18 14:20:30.193578 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.193559 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz_b5470b4e-48bc-4c6e-a8a2-ada8d7636789/util/0.log" Jul 18 14:20:30.424032 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.424000 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6ndkm_d2994a17-3a4b-402f-9d08-8a420276497e/manager/0.log" Jul 18 14:20:30.526430 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.526394 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tmnxn_3998ee96-6963-47d3-9d14-c52a002aa7ad/manager/0.log" Jul 18 14:20:30.631618 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.631534 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rvf62_4b4d5ac0-f879-47b3-95ca-a6ce4d34d470/kuadrant-console-plugin/0.log" Jul 18 14:20:30.740170 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.740126 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cmsnz_30cae6b3-f118-4cee-af3b-e82bf1dbb6be/registry-server/0.log" Jul 18 14:20:30.962320 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:30.962299 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tw58w_7f69186e-c708-4b22-8fa8-4f0e622ece75/limitador/0.log" Jul 18 14:20:31.076438 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:31.076406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dscqk_1431c65a-db6e-4be3-acb1-fce3df414ec7/manager/0.log" Jul 18 14:20:31.726133 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:31.726099 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65948f8d-qvnzv_94017353-c705-44dc-b6df-380aa7dbd780/kube-auth-proxy/0.log" Jul 18 14:20:31.835616 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:31.835586 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/1.log" Jul 18 14:20:31.946208 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:31.946177 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frbgv_e2765f9f-2e3a-4810-a3e3-52afab8057ed/payload-pre-processing/0.log" Jul 18 14:20:32.057031 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:32.056953 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kpt67_5852dde2-e6ec-4b49-a844-d58db0e9a726/payload-processing/0.log" Jul 18 14:20:32.610004 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:32.609973 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dk7z7_2ace6b59-bd73-4c9c-96a7-175de649ad36/main/0.log" Jul 18 14:20:38.382584 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382548 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq9sq/must-gather-dr229"] Jul 18 14:20:38.382988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382935 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" Jul 18 14:20:38.382988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382945 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" Jul 18 14:20:38.382988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382967 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" containerName="test-curl-7648-139623983511856" Jul 18 14:20:38.382988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382974 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" containerName="test-curl-7648-139623983511856" Jul 18 14:20:38.382988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382982 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" Jul 18 14:20:38.382988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.382987 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" Jul 18 14:20:38.383182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.383043 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" Jul 18 14:20:38.383182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.383053 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c74bb2e-a7b0-48d9-8c81-dfd3bbd737de" containerName="istio-proxy" Jul 18 14:20:38.383182 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.383060 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9851c4-ba33-4e13-8191-95d3cbc13ae8" containerName="test-curl-7648-139623983511856" Jul 18 14:20:38.388134 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.388109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.390929 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.390905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq9sq\"/\"default-dockercfg-r9wfw\"" Jul 18 14:20:38.391054 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.390910 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq9sq\"/\"openshift-service-ca.crt\"" Jul 18 14:20:38.391054 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.390970 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq9sq\"/\"kube-root-ca.crt\"" Jul 18 14:20:38.396463 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.396440 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq9sq/must-gather-dr229"] Jul 18 14:20:38.539207 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.539172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78wj\" (UniqueName: \"kubernetes.io/projected/1887b909-9684-411c-a404-1aa8b9a07ef1-kube-api-access-q78wj\") pod \"must-gather-dr229\" (UID: \"1887b909-9684-411c-a404-1aa8b9a07ef1\") " pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.539417 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.539227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1887b909-9684-411c-a404-1aa8b9a07ef1-must-gather-output\") pod \"must-gather-dr229\" (UID: \"1887b909-9684-411c-a404-1aa8b9a07ef1\") " pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.640417 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.640298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1887b909-9684-411c-a404-1aa8b9a07ef1-must-gather-output\") pod \"must-gather-dr229\" (UID: \"1887b909-9684-411c-a404-1aa8b9a07ef1\") " pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.640582 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.640454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78wj\" (UniqueName: \"kubernetes.io/projected/1887b909-9684-411c-a404-1aa8b9a07ef1-kube-api-access-q78wj\") pod \"must-gather-dr229\" (UID: \"1887b909-9684-411c-a404-1aa8b9a07ef1\") " pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.640665 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.640651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1887b909-9684-411c-a404-1aa8b9a07ef1-must-gather-output\") pod \"must-gather-dr229\" (UID: \"1887b909-9684-411c-a404-1aa8b9a07ef1\") " pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.655745 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.655713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78wj\" (UniqueName: \"kubernetes.io/projected/1887b909-9684-411c-a404-1aa8b9a07ef1-kube-api-access-q78wj\") pod \"must-gather-dr229\" (UID: \"1887b909-9684-411c-a404-1aa8b9a07ef1\") " pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.699703 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.699663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq9sq/must-gather-dr229" Jul 18 14:20:38.839015 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:38.838986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq9sq/must-gather-dr229"] Jul 18 14:20:38.843421 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:20:38.842629 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1887b909_9684_411c_a404_1aa8b9a07ef1.slice/crio-580f49542c9d5f51b4247fede7f55094bfcdbe597fa9778a05319d8b333ea4ab WatchSource:0}: Error finding container 580f49542c9d5f51b4247fede7f55094bfcdbe597fa9778a05319d8b333ea4ab: Status 404 returned error can't find the container with id 580f49542c9d5f51b4247fede7f55094bfcdbe597fa9778a05319d8b333ea4ab Jul 18 14:20:39.351120 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:39.351079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq9sq/must-gather-dr229" event={"ID":"1887b909-9684-411c-a404-1aa8b9a07ef1","Type":"ContainerStarted","Data":"580f49542c9d5f51b4247fede7f55094bfcdbe597fa9778a05319d8b333ea4ab"} Jul 18 14:20:40.361436 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:40.361401 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq9sq/must-gather-dr229" event={"ID":"1887b909-9684-411c-a404-1aa8b9a07ef1","Type":"ContainerStarted","Data":"6f1150d9025c3bd61513ec4e55f75cf4e5c69e42414ee1e7fcccf923358afb2c"} Jul 18 14:20:40.361929 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:40.361454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq9sq/must-gather-dr229" event={"ID":"1887b909-9684-411c-a404-1aa8b9a07ef1","Type":"ContainerStarted","Data":"b1d1421f95446860bd792614521fa7e405a3498c342c4adb4ad1bc8718981e07"} Jul 18 14:20:40.380166 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:40.380097 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq9sq/must-gather-dr229" podStartSLOduration=1.537076669 podStartE2EDuration="2.380075247s" totalImagesPullingTime="842.998578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 14:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:20:40.376434115 +0000 UTC m=+2890.925998797" watchObservedRunningTime="2026-07-18 14:20:40.380075247 +0000 UTC m=+2890.929639929" Jul 18 14:20:41.244576 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:41.244541 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ctdfr_bf9097cb-304f-4a1a-85f3-f60bd96eddc9/global-pull-secret-syncer/0.log" Jul 18 14:20:41.392021 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:41.391985 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mvnm2_889d03f4-e9aa-4b62-bc1b-fd2f77093382/konnectivity-agent/0.log" Jul 18 14:20:41.457908 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:41.457881 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-156.ec2.internal_d7a9cf36e12cd844e88b9ba2474ad903/haproxy/0.log" Jul 18 14:20:45.819758 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:45.819719 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9_b345f19e-eadb-4cfa-b1f4-e76c18f3257a/extract/0.log" Jul 18 14:20:45.857188 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:45.857159 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9_b345f19e-eadb-4cfa-b1f4-e76c18f3257a/util/0.log" Jul 18 14:20:45.897020 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:45.896986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rx6d9_b345f19e-eadb-4cfa-b1f4-e76c18f3257a/pull/0.log" Jul 18 14:20:45.946473 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:45.946434 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c_79eff395-f999-40bc-96cb-164545194b03/extract/0.log" Jul 18 14:20:45.967479 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:45.967451 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c_79eff395-f999-40bc-96cb-164545194b03/util/0.log" Jul 18 14:20:45.990753 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:45.990726 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0csv9c_79eff395-f999-40bc-96cb-164545194b03/pull/0.log" Jul 18 14:20:46.022627 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.022593 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29_72b7aeca-8a94-4994-8f83-1f2061afe4c1/extract/0.log" Jul 18 14:20:46.045244 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.045217 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29_72b7aeca-8a94-4994-8f83-1f2061afe4c1/util/0.log" Jul 18 14:20:46.066385 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.066346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738df29_72b7aeca-8a94-4994-8f83-1f2061afe4c1/pull/0.log" Jul 18 14:20:46.095508 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.095323 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz_b5470b4e-48bc-4c6e-a8a2-ada8d7636789/extract/0.log" Jul 18 14:20:46.116038 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.115975 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz_b5470b4e-48bc-4c6e-a8a2-ada8d7636789/util/0.log" Jul 18 14:20:46.136101 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.136074 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1l4ftz_b5470b4e-48bc-4c6e-a8a2-ada8d7636789/pull/0.log" Jul 18 14:20:46.794640 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.794601 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-6ndkm_d2994a17-3a4b-402f-9d08-8a420276497e/manager/0.log" Jul 18 14:20:46.820692 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.820658 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tmnxn_3998ee96-6963-47d3-9d14-c52a002aa7ad/manager/0.log" Jul 18 14:20:46.856249 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.856202 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rvf62_4b4d5ac0-f879-47b3-95ca-a6ce4d34d470/kuadrant-console-plugin/0.log" Jul 18 14:20:46.901859 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:46.901823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cmsnz_30cae6b3-f118-4cee-af3b-e82bf1dbb6be/registry-server/0.log" Jul 18 14:20:47.126628 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:47.126512 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tw58w_7f69186e-c708-4b22-8fa8-4f0e622ece75/limitador/0.log" Jul 18 14:20:47.224547 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:47.224516 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dscqk_1431c65a-db6e-4be3-acb1-fce3df414ec7/manager/0.log" Jul 18 14:20:48.934197 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:48.934147 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f9c5745b-bmd45_93549ecb-87e4-4307-9066-1e6a03b62ad0/metrics-server/0.log" Jul 18 14:20:49.090856 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.090822 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cjhrl_46213551-ca76-4a31-b772-31bc75914bf6/node-exporter/0.log" Jul 18 14:20:49.125831 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.125752 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cjhrl_46213551-ca76-4a31-b772-31bc75914bf6/kube-rbac-proxy/0.log" Jul 18 14:20:49.147878 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.147784 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cjhrl_46213551-ca76-4a31-b772-31bc75914bf6/init-textfile/0.log" Jul 18 14:20:49.248360 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.248328 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-npfs6_fde10a60-5f25-41d1-8845-8fcc4de3798d/kube-rbac-proxy-main/0.log" Jul 18 14:20:49.272951 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.272925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-npfs6_fde10a60-5f25-41d1-8845-8fcc4de3798d/kube-rbac-proxy-self/0.log" Jul 18 14:20:49.292565 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.292517 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-npfs6_fde10a60-5f25-41d1-8845-8fcc4de3798d/openshift-state-metrics/0.log" Jul 18 14:20:49.512550 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.512509 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-7lx95_47ebf2ff-7da4-43dd-b4cb-3298f2963d3a/prometheus-operator-admission-webhook/0.log" Jul 18 14:20:49.554815 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.554787 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f967d76d-7749t_ea6d35ad-7f12-43a8-bd7b-63a434560133/telemeter-client/0.log" Jul 18 14:20:49.580427 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.580396 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f967d76d-7749t_ea6d35ad-7f12-43a8-bd7b-63a434560133/reload/0.log" Jul 18 14:20:49.600967 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:49.600939 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55f967d76d-7749t_ea6d35ad-7f12-43a8-bd7b-63a434560133/kube-rbac-proxy/0.log" Jul 18 14:20:51.833676 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.833646 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx"] Jul 18 14:20:51.839334 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.839304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:51.851271 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.850901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx"] Jul 18 14:20:51.901400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.900888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-proc\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:51.901400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.901023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-lib-modules\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:51.901400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.901095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-sys\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:51.901400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.901121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf62j\" (UniqueName: \"kubernetes.io/projected/099d9fe8-b1ac-40b8-8515-f672ac68d084-kube-api-access-cf62j\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:51.901400 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:51.901155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-podres\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002175 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-proc\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002389 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-lib-modules\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002389 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-proc\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002389 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-sys\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002530 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf62j\" (UniqueName: \"kubernetes.io/projected/099d9fe8-b1ac-40b8-8515-f672ac68d084-kube-api-access-cf62j\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002530 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-podres\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002530 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-lib-modules\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002530 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-sys\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.002679 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.002563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/099d9fe8-b1ac-40b8-8515-f672ac68d084-podres\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.013889 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.013855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf62j\" (UniqueName: \"kubernetes.io/projected/099d9fe8-b1ac-40b8-8515-f672ac68d084-kube-api-access-cf62j\") pod \"perf-node-gather-daemonset-lq9wx\" (UID: \"099d9fe8-b1ac-40b8-8515-f672ac68d084\") " pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.146337 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.146253 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fd8c85fc8-2kwrs_64d2798c-7479-4a95-983e-829c2f79831b/console/0.log" Jul 18 14:20:52.162276 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.162246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:52.375125 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.375014 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx"] Jul 18 14:20:52.383878 ip-10-0-136-156 kubenswrapper[2564]: W0718 14:20:52.383829 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod099d9fe8_b1ac_40b8_8515_f672ac68d084.slice/crio-9b6fb91cd484c4e90acf81392517737c033abfdd43d8e11b1c556f810fc7dc72 WatchSource:0}: Error finding container 9b6fb91cd484c4e90acf81392517737c033abfdd43d8e11b1c556f810fc7dc72: Status 404 returned error can't find the container with id 9b6fb91cd484c4e90acf81392517737c033abfdd43d8e11b1c556f810fc7dc72 Jul 18 14:20:52.483189 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:52.483151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" event={"ID":"099d9fe8-b1ac-40b8-8515-f672ac68d084","Type":"ContainerStarted","Data":"9b6fb91cd484c4e90acf81392517737c033abfdd43d8e11b1c556f810fc7dc72"} Jul 18 14:20:53.494622 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:53.494573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" event={"ID":"099d9fe8-b1ac-40b8-8515-f672ac68d084","Type":"ContainerStarted","Data":"4819fb083ebde62f40cf428c34145473f383c834543cfb48155bc27cbbb5a9d2"} Jul 18 14:20:53.495203 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:53.494664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:53.511988 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:53.511922 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" podStartSLOduration=2.511899707 podStartE2EDuration="2.511899707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 14:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 14:20:53.50943057 +0000 UTC m=+2904.058995255" watchObservedRunningTime="2026-07-18 14:20:53.511899707 +0000 UTC m=+2904.061464390" Jul 18 14:20:53.878729 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:53.878623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9pcn_2e9d48b8-6612-4efd-9c5c-34bbaa70457e/dns/0.log" Jul 18 14:20:53.896362 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:53.896334 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9pcn_2e9d48b8-6612-4efd-9c5c-34bbaa70457e/kube-rbac-proxy/0.log" Jul 18 14:20:53.958474 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:53.958445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gk8qd_30288396-d289-4c60-a4d1-4da17345d284/dns-node-resolver/0.log" Jul 18 14:20:54.470570 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:54.470539 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78b5b6cc4c-vc4xx_76a4b88f-fa98-401e-9698-74492b1da989/registry/0.log" Jul 18 14:20:54.531758 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:54.531732 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ms9b7_2923f5c6-d83f-4b2f-a161-cd736a53bce4/node-ca/0.log" Jul 18 14:20:55.574390 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:55.574355 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65948f8d-qvnzv_94017353-c705-44dc-b6df-380aa7dbd780/kube-auth-proxy/0.log" Jul 18 14:20:55.602811 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:55.602782 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/0.log" Jul 18 14:20:55.637098 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:55.637062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/1.log" Jul 18 14:20:55.663863 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:55.663825 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frbgv_e2765f9f-2e3a-4810-a3e3-52afab8057ed/payload-pre-processing/0.log" Jul 18 14:20:55.690650 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:55.690619 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kpt67_5852dde2-e6ec-4b49-a844-d58db0e9a726/payload-processing/0.log" Jul 18 14:20:56.404858 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:56.404829 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmz7z_31e00d6b-d536-4001-8049-cb466a6257df/serve-healthcheck-canary/0.log" Jul 18 14:20:56.929019 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:56.928980 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-25r94_ceebf852-8a47-433c-b42a-b0191bcb04ac/kube-rbac-proxy/0.log" Jul 18 14:20:56.948314 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:56.948286 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-25r94_ceebf852-8a47-433c-b42a-b0191bcb04ac/exporter/0.log" Jul 18 14:20:56.967074 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:56.967045 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-25r94_ceebf852-8a47-433c-b42a-b0191bcb04ac/extractor/0.log" Jul 18 14:20:59.181780 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:59.181752 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pfrd7_1d32444b-58be-4066-b0d0-bd3451756721/manager/0.log" Jul 18 14:20:59.517275 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:59.516329 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq9sq/perf-node-gather-daemonset-lq9wx" Jul 18 14:20:59.621847 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:59.621820 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w8txs_fac34956-9721-4e24-9f84-87fde2c723f7/server/0.log" Jul 18 14:20:59.711996 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:59.711960 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mcpjl_0492ceb8-72fc-43ea-b95b-45c657ff946d/manager/0.log" Jul 18 14:20:59.865866 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:59.865794 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7687b4d-shppk_d179a8f6-f972-46cc-83b1-409934056e12/manager/0.log" Jul 18 14:20:59.884342 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:20:59.884311 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7bd7687b4d-shppk_d179a8f6-f972-46cc-83b1-409934056e12/copy-manifests/0.log" Jul 18 14:21:01.297049 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:01.296971 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lvkr5_a322fc9d-ddac-4d81-a206-66f305c66a45/openshift-lws-operator/0.log" Jul 18 14:21:02.172833 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:02.172796 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65948f8d-qvnzv_94017353-c705-44dc-b6df-380aa7dbd780/kube-auth-proxy/0.log" Jul 18 14:21:02.190348 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:02.190307 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/0.log" Jul 18 14:21:02.224281 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:02.224244 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/1.log" Jul 18 14:21:02.243651 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:02.243621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frbgv_e2765f9f-2e3a-4810-a3e3-52afab8057ed/payload-pre-processing/0.log" Jul 18 14:21:02.261740 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:02.261708 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kpt67_5852dde2-e6ec-4b49-a844-d58db0e9a726/payload-processing/0.log" Jul 18 14:21:03.569950 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:03.569876 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65948f8d-qvnzv_94017353-c705-44dc-b6df-380aa7dbd780/kube-auth-proxy/0.log" Jul 18 14:21:03.593306 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:03.593265 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/0.log" Jul 18 14:21:03.630595 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:03.630562 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bk579_0c4a5666-4fc5-4262-9cc9-56679b6d3c39/istio-proxy/1.log" Jul 18 14:21:03.656547 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:03.656516 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frbgv_e2765f9f-2e3a-4810-a3e3-52afab8057ed/payload-pre-processing/0.log" Jul 18 14:21:03.676234 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:03.676203 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-kpt67_5852dde2-e6ec-4b49-a844-d58db0e9a726/payload-processing/0.log" Jul 18 14:21:07.838027 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.837999 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/kube-multus-additional-cni-plugins/0.log" Jul 18 14:21:07.857207 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.857178 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/egress-router-binary-copy/0.log" Jul 18 14:21:07.875851 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.875823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/cni-plugins/0.log" Jul 18 14:21:07.894303 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.894276 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/bond-cni-plugin/0.log" Jul 18 14:21:07.913776 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.913751 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/routeoverride-cni/0.log" Jul 18 14:21:07.931867 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.931844 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/whereabouts-cni-bincopy/0.log" Jul 18 14:21:07.949602 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:07.949575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bxqcb_ee9705d3-8a04-41a6-8b60-eeeb0304ead9/whereabouts-cni/0.log" Jul 18 14:21:08.155920 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:08.155844 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r6pzv_58e3f636-d6a9-4a7c-910f-991065c5b0e0/kube-multus/0.log" Jul 18 14:21:08.254848 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:08.254816 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wbwm2_8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe/network-metrics-daemon/0.log" Jul 18 14:21:08.270998 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:08.270970 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wbwm2_8cb91a5c-b9f8-42cc-85d6-dbcbef7336fe/kube-rbac-proxy/0.log" Jul 18 14:21:09.064500 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.064415 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/ovn-controller/0.log" Jul 18 14:21:09.078665 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.078633 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/ovn-acl-logging/0.log" Jul 18 14:21:09.101405 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.101360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/kube-rbac-proxy-node/0.log" Jul 18 14:21:09.126744 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.126714 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/kube-rbac-proxy-ovn-metrics/0.log" Jul 18 14:21:09.141653 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.141621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/northd/0.log" Jul 18 14:21:09.160487 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.160441 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/nbdb/0.log" Jul 18 14:21:09.178305 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.178282 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/sbdb/0.log" Jul 18 14:21:09.243051 ip-10-0-136-156 kubenswrapper[2564]: I0718 14:21:09.243019 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rjnf_06575fd3-b79e-4112-a6a8-dbd3e5c22a4b/ovnkube-controller/0.log"