Jul 19 14:57:27.475091 ip-10-0-140-163 systemd[1]: Starting Kubernetes Kubelet... Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.936293 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941917 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941929 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941932 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941935 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941940 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941943 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941946 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:28.024085 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941950 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:28.017969 ip-10-0-140-163 systemd[1]: Started Kubernetes Kubelet. Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941953 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941956 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941958 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941961 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941964 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941967 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941970 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941973 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941975 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941978 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941986 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941989 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941992 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941994 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.941997 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942000 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942002 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942005 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942008 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:28.274507 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942011 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942014 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942017 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942019 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942022 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942024 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942027 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942029 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942032 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942035 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942037 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942040 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942043 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942045 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942048 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942051 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942054 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942057 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942059 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:28.280838 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942062 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942065 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942067 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942070 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942072 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942075 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942078 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942080 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942083 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942088 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942090 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942093 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942096 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942099 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942101 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942104 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942108 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942111 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942114 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:28.282422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942117 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942120 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942122 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942125 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942127 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942130 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942132 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942135 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942137 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942140 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942143 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942146 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942148 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942151 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942153 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942156 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942159 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942161 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942167 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942170 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:28.283655 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942173 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942176 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942179 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942182 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942185 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942188 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942191 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942194 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942197 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942199 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942202 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942205 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942209 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942213 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942904 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942911 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942914 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942917 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942920 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942923 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:28.288547 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942926 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942929 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942931 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942934 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942938 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942941 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942943 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942946 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942949 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942952 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942954 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942958 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942960 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942963 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942966 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942969 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942971 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942974 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942977 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942980 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:28.289212 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942983 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942985 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942990 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942993 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942996 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.942999 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943002 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943005 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943008 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943011 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943013 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943016 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943018 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943021 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943024 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943026 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943029 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943034 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943037 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:28.289917 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943040 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943042 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943045 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943049 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943053 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943056 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943059 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943061 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943064 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943067 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943069 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943072 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943075 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943077 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943080 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943083 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943086 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943089 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943091 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:28.290526 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943094 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943096 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943099 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943101 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943104 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943106 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943109 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943111 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943114 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943117 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943119 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943125 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943127 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943130 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943132 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943135 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943139 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943143 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943145 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943148 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:28.291103 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943150 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943153 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943156 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943160 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943163 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943165 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943168 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943170 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943173 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943176 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943178 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943181 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943183 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943186 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943189 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943550 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943554 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943557 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943560 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943563 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:28.291710 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943565 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943568 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943571 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943577 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943580 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943582 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943585 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943588 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943591 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943594 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943597 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943600 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943602 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943605 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943607 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943610 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943612 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943615 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943617 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943620 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:28.292331 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943622 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943625 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943628 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943630 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943633 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943635 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943638 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943640 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943643 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943645 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943648 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943652 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943655 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943658 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943661 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943666 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943670 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943672 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943676 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943678 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:28.292961 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943681 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943683 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943686 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943688 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943691 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943694 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943697 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943699 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943702 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943704 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943707 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943709 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943712 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943714 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943717 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943720 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943722 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943725 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943727 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943730 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:28.293528 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943732 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943735 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943738 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943740 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943743 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943747 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943750 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943754 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943757 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943760 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943762 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943765 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943768 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943770 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943773 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943776 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943779 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943781 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943800 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943803 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:28.294245 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943806 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943809 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943811 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943814 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943818 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943822 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943825 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943828 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943831 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943834 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943837 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943840 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943843 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.943846 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943931 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943938 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943944 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943948 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943952 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943957 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 19 14:57:28.294811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943962 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943966 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943970 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943973 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943977 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943980 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943983 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943986 2580 flags.go:64] FLAG: --cgroup-root="" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943989 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943993 2580 flags.go:64] FLAG: --client-ca-file="" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943995 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.943998 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944004 2580 flags.go:64] FLAG: --cluster-domain="" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944007 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944010 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944013 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944017 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944020 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944023 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944027 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944030 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944033 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944036 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944039 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 19 14:57:28.295383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944042 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944045 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944049 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944052 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944055 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944057 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944061 2580 flags.go:64] FLAG: --enable-server="true" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944065 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944070 2580 flags.go:64] FLAG: --event-burst="100" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944074 2580 flags.go:64] FLAG: --event-qps="50" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944077 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944080 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944084 2580 flags.go:64] FLAG: --eviction-hard="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944088 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944090 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944094 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944097 2580 flags.go:64] FLAG: --eviction-soft="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944100 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944103 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944112 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944116 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944120 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944123 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944126 2580 flags.go:64] FLAG: --feature-gates="" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944130 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 19 14:57:28.296004 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944133 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944136 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944139 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944143 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944146 2580 flags.go:64] FLAG: --help="false" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944149 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944152 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944155 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944158 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944161 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944165 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944168 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944173 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944176 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944180 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944183 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944186 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944189 2580 flags.go:64] FLAG: --kube-reserved="" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944192 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944196 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944199 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944202 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944205 2580 flags.go:64] FLAG: --lock-file="" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944208 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 19 14:57:28.296773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944211 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944214 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944220 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944223 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944226 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944228 2580 flags.go:64] FLAG: --logging-format="text" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944231 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944235 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944238 2580 flags.go:64] FLAG: --manifest-url="" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944241 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944245 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944248 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944252 2580 flags.go:64] FLAG: --max-pods="110" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944255 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944258 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944260 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944263 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944266 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944269 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944272 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944281 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944284 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944288 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 19 14:57:28.297439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944291 2580 flags.go:64] FLAG: --pod-cidr="" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944294 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944299 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944302 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944306 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944309 2580 flags.go:64] FLAG: --port="10250" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944312 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944315 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a09225f15555eba7" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944318 2580 flags.go:64] FLAG: --qos-reserved="" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944321 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944324 2580 flags.go:64] FLAG: --register-node="true" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944327 2580 flags.go:64] FLAG: --register-with-taints="" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944331 2580 flags.go:64] FLAG: --registry-burst="10" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944333 2580 flags.go:64] FLAG: --registry-qps="5" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944336 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944339 2580 flags.go:64] FLAG: --reserved-memory="" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944342 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944346 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944348 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944351 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944354 2580 flags.go:64] FLAG: --runonce="false" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944357 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944360 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944363 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944366 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 19 14:57:28.298124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944369 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944371 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944375 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944378 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944382 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944385 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944388 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944392 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944395 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944398 2580 flags.go:64] FLAG: --system-cgroups="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944401 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944406 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944409 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944412 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944417 2580 flags.go:64] FLAG: --tls-min-version="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944420 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944423 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944426 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944429 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944431 2580 flags.go:64] FLAG: --v="2" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944435 2580 flags.go:64] FLAG: --version="false" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944439 2580 flags.go:64] FLAG: --vmodule="" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944443 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.944446 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944558 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:28.299087 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944562 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944565 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944568 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944571 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944573 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944576 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944578 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944581 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944584 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944587 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944589 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944593 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944595 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944598 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944602 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944604 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944609 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944613 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944616 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:28.299779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944620 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944622 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944626 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944628 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944631 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944634 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944637 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944640 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944642 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944645 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944647 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944652 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944656 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944659 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944662 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944664 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944667 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944669 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944672 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:28.453538 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944675 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944677 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944680 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944683 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944685 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944689 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944691 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944694 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944698 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944701 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944703 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944706 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944709 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944711 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944714 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944716 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944719 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944722 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944725 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944727 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:28.454137 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944730 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944733 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944735 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944738 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944741 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944743 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944746 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944749 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944751 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944754 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944757 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944759 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944762 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944764 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944767 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944769 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944772 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944776 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944778 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944781 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:28.454677 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944800 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944803 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944815 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944817 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944820 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944823 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944826 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944828 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944831 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944835 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944837 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944840 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944842 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944845 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944848 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944850 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944853 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944856 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944858 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:28.455278 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.944861 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.946397 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.953520 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.953539 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.953559 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.953565 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953619 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953624 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953628 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953631 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953635 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953637 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953640 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953643 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953646 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:28.455803 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953649 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953652 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953655 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953657 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953661 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953665 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953668 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953672 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953676 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953682 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953685 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953688 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953692 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953695 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953698 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953700 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953703 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953705 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953708 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:28.456267 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953711 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953714 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953717 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953719 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953722 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953724 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953727 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953730 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953733 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953735 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953738 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953740 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953743 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953745 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953748 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953750 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953753 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953755 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953758 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953760 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:28.456781 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953763 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953765 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953769 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953772 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953774 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953778 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953780 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953798 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953801 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953804 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953806 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953809 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953812 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953815 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953818 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953821 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953824 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953827 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953829 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953832 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:28.457353 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953835 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953837 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953841 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953844 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953847 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953850 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953853 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953856 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953859 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953861 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953864 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953866 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953869 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953872 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953876 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953878 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953881 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953885 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953888 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:28.457915 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953890 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953893 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953895 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953898 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953901 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953903 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953906 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953908 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953912 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953915 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953917 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.953920 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.953925 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954037 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:28.458437 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954043 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954047 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954051 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954054 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954058 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954061 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954064 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954066 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954069 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954072 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954074 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954077 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954080 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954083 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954087 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954089 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954093 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954096 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954099 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:28.458845 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954101 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954104 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954106 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954109 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954111 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954114 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954116 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954119 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954122 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954125 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954127 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954130 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954132 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954135 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954137 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954140 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954142 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954145 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954148 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954150 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:28.459395 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954153 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954155 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954158 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954160 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954163 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954165 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954168 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954171 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954174 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954178 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954181 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954183 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954186 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954190 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954193 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954197 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954199 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954202 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954205 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:28.459944 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954208 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954210 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954214 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954216 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954219 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954221 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954224 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954227 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954229 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954232 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954234 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954237 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954239 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954242 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954244 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954247 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954249 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954252 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954254 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954257 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:28.460520 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954261 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954264 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954268 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954270 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954273 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954276 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954278 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954281 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954283 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954286 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954288 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954291 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954293 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954295 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954298 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954301 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954304 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954306 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954309 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:28.461088 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:27.954311 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.954316 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.954426 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.959545 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.961380 2580 server.go:1019] "Starting client certificate rotation" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.961478 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.961518 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.964144 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.965305 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.975760 2580 log.go:25] "Validated CRI v1 runtime API" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.978233 2580 log.go:25] "Validated CRI v1 image API" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.979034 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 19 14:57:28.461624 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.982752 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 85eadd97-9c26-4195-849a-4cf214f309aa:/dev/nvme0n1p3 a88bfe3d-c225-4861-b385-b6d052e16114:/dev/nvme0n1p4] Jul 19 14:57:28.462000 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.982776 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 19 14:57:28.462000 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.989178 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.990141 2580 manager.go:216] Machine: {Timestamp:2026-07-19 14:57:27.988476361 +0000 UTC m=+0.366546879 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101779 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a04d923c5d0185ce9984bf62258ab SystemUUID:ec2a04d9-23c5-d018-5ce9-984bf62258ab BootID:65d5419b-bb35-4eba-b5fb-71deba7faabb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:23:90:42:02:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:23:90:42:02:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:5e:93:04:e6:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.990274 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.990380 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.993944 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.993974 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.994825 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.994836 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.994847 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.994861 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:27.996849 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000469 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000574 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000600 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000610 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000623 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000632 2580 kubelet.go:398] "Adding apiserver pod source" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.000641 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 19 14:57:28.462168 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.001126 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.001142 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.008800 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.011393 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.011433 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.011889 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.011889 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012852 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012865 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012871 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012877 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012886 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012892 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012901 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012906 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012913 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012919 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012929 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.012934 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.013836 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.013847 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.017182 2580 server.go:1268] "Started kubelet" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.017314 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.017387 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.017458 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.019195 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.020310 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.024762 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.026628 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.026711 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.ec2.internal.18c3b8c85a522240 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.ec2.internal,UID:ip-10-0-140-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.ec2.internal,},FirstTimestamp:2026-07-19 14:57:28.017150528 +0000 UTC m=+0.395221049,LastTimestamp:2026-07-19 14:57:28.017150528 +0000 UTC m=+0.395221049,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.ec2.internal,}" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.028851 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.028872 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.028948 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029514 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029605 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029620 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029691 2580 factory.go:55] Registering systemd factory Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029724 2580 factory.go:223] Registration of the systemd container factory successfully Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029727 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029736 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029935 2580 factory.go:153] Registering CRI-O factory Jul 19 14:57:28.462759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029948 2580 factory.go:223] Registration of the crio container factory successfully Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.029995 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.030016 2580 factory.go:103] Registering Raw factory Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.030030 2580 manager.go:1196] Started watching for new ooms in manager Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.030465 2580 manager.go:349] Starting recovery of all containers Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.034066 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.034117 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.035400 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ql6c4" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.040664 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ql6c4" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.041204 2580 manager.go:354] Recovery completed Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.047873 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050133 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050158 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050173 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050691 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050701 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050716 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.050701 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.051421 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.051446 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.051459 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.051610 2580 policy_none.go:49] "None policy: Start" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.051622 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.051631 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.052090 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.ec2.internal.18c3b8c85c499947 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.ec2.internal,UID:ip-10-0-140-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.ec2.internal,},FirstTimestamp:2026-07-19 14:57:28.050145607 +0000 UTC m=+0.428216124,LastTimestamp:2026-07-19 14:57:28.050145607 +0000 UTC m=+0.428216124,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.ec2.internal,}" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.053586 2580 policy_none.go:47] "Start" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.129712 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.135519 2580 manager.go:337] "Starting Device Plugin manager" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.135555 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.135565 2580 server.go:87] "Starting device plugin registration server" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.135861 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.135874 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.135953 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.136044 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.136053 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.136509 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.136555 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.140510 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.141747 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 19 14:57:28.464167 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.141780 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.141820 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.141858 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.143659 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.235972 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.236909 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.236943 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.236961 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.236989 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.239552 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-163.ec2.internal\" not found" node="ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.241933 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"] Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.241994 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.245426 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.245452 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.245463 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.245511 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.245528 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-163.ec2.internal\": node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.246575 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.246705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.246730 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.247262 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.247280 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.247287 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.247297 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.247301 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.247305 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.248274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.248297 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.248979 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.249009 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.249023 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.267873 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.271306 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.ec2.internal\" not found" node="ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.276663 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.ec2.internal\" not found" node="ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.332296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ef091f26c73031458c515489d9e0ec4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"4ef091f26c73031458c515489d9e0ec4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.625833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.332480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ef091f26c73031458c515489d9e0ec4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"4ef091f26c73031458c515489d9e0ec4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.332498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a6110787fb36561c0436ce6584028c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.ec2.internal\" (UID: \"b4a6110787fb36561c0436ce6584028c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.368704 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.433033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ef091f26c73031458c515489d9e0ec4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"4ef091f26c73031458c515489d9e0ec4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.433071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ef091f26c73031458c515489d9e0ec4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"4ef091f26c73031458c515489d9e0ec4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.433092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ef091f26c73031458c515489d9e0ec4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"4ef091f26c73031458c515489d9e0ec4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.433125 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a6110787fb36561c0436ce6584028c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.ec2.internal\" (UID: \"b4a6110787fb36561c0436ce6584028c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.433088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b4a6110787fb36561c0436ce6584028c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.ec2.internal\" (UID: \"b4a6110787fb36561c0436ce6584028c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.433118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ef091f26c73031458c515489d9e0ec4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"4ef091f26c73031458c515489d9e0ec4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.469325 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.569926 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.575083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:28.629903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:28.580945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" Jul 19 14:57:30.291845 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:28.671061 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:30.291845 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.285350 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.385827 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.397945 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.404054 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.443561 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.456688 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.457566 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" Jul 19 14:57:30.465849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.463479 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.474615 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.474742 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.474778 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.515272 2580 apiserver.go:52] "Watching apiserver" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.522299 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.522855 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xpbn5","openshift-network-operator/iptables-alerter-xqbcl","kube-system/global-pull-secret-syncer-rgb9j","openshift-cluster-node-tuning-operator/tuned-49djs","openshift-dns/node-resolver-7k7rd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal","openshift-multus/multus-additional-cni-plugins-hcjp9","openshift-network-diagnostics/network-check-target-gsvrk","openshift-ovn-kubernetes/ovnkube-node-h4h72","kube-system/konnectivity-agent-mx6jp","kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w","openshift-image-registry/node-ca-mscg5","openshift-multus/multus-c689m"] Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.524162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.524227 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.525546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.526565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c689m" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.530938 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.530989 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.531224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.531257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-59kt9\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.531488 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.531806 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.531816 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.532053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.532483 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.532502 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4c9qp\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.533412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.534763 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.535043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.535081 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.535162 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.535196 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxwfw\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.535484 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.535965 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.536155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.536183 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bq7zq\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.537311 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzdkc\"" Jul 19 14:57:30.642632 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.537404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.537670 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.537738 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538066 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538100 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xvqbp\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538115 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538141 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538178 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538311 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.538651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.539548 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.539615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qntgh\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.539657 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.540004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.540074 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.540488 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.540557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.540732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2z896\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.541462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.543134 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.543181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.543242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.546946 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.546997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.547064 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.547087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qknzm\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.547119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-sys-fs\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vt27\" (UniqueName: \"kubernetes.io/projected/c34f641b-e476-47c0-9dce-506e0d8b2c36-kube-api-access-9vt27\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.644384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f37c79b-1c46-4400-81a3-c64d9a2384c3-iptables-alerter-script\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-k8s-cni-cncf-io\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-cni-bin\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-systemd-units\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-slash\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b035e097-e89a-46e2-ac9e-83543a280851-cni-binary-copy\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-var-lib-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-cni-netd\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysctl-d\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgrs\" (UniqueName: \"kubernetes.io/projected/a4142486-9667-4b0c-9304-d6edc4c4d09b-kube-api-access-7kgrs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-device-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-os-release\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e81ea0-1dea-466e-a980-176b3a996a7d-tmp-dir\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.645264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-kubernetes\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/266b3d70-24cc-469e-8906-d6822c6beaa7-agent-certs\") pod \"konnectivity-agent-mx6jp\" (UID: \"266b3d70-24cc-469e-8906-d6822c6beaa7\") " pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550483 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovnkube-script-lib\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63cfb73d-a2d1-41a6-8911-03772f734c2d-dbus\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f37c79b-1c46-4400-81a3-c64d9a2384c3-host-slash\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-os-release\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-kubelet\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-etc-kubernetes\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysconfig\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghm8s\" (UniqueName: \"kubernetes.io/projected/754bd0cf-54de-4f3f-b513-501ec601ec40-kube-api-access-ghm8s\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-netns\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-cnibin\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-etc-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.645987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-tuned\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwh7\" (UniqueName: \"kubernetes.io/projected/ae881e3e-1525-4350-9db6-90a307e44924-kube-api-access-slwh7\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-cni-multus\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dc6h\" (UniqueName: \"kubernetes.io/projected/b035e097-e89a-46e2-ac9e-83543a280851-kube-api-access-8dc6h\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-ovn\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-systemd\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djhm\" (UniqueName: \"kubernetes.io/projected/c1e81ea0-1dea-466e-a980-176b3a996a7d-kube-api-access-7djhm\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b035e097-e89a-46e2-ac9e-83543a280851-multus-daemon-config\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.550986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-sys\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-lib-modules\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36be97b5-0bc8-41ad-847e-e8b817a9cf46-tmp\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-system-cni-dir\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-hostroot\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-systemd\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.646549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvqm\" (UniqueName: \"kubernetes.io/projected/88f133f4-9d7a-4622-8eb8-97c9a630fbde-kube-api-access-7dvqm\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-run\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-socket-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-cni-binary-copy\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-system-cni-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovnkube-config\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-registration-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1e81ea0-1dea-466e-a980-176b3a996a7d-hosts-file\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-log-socket\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovn-node-metrics-cert\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zcjm\" (UniqueName: \"kubernetes.io/projected/36be97b5-0bc8-41ad-847e-e8b817a9cf46-kube-api-access-6zcjm\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/754bd0cf-54de-4f3f-b513-501ec601ec40-host\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rps8f\" (UniqueName: \"kubernetes.io/projected/9f37c79b-1c46-4400-81a3-c64d9a2384c3-kube-api-access-rps8f\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.647150 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-conf-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-node-log\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-cni-bin\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-host\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/266b3d70-24cc-469e-8906-d6822c6beaa7-konnectivity-ca\") pod \"konnectivity-agent-mx6jp\" (UID: \"266b3d70-24cc-469e-8906-d6822c6beaa7\") " pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/754bd0cf-54de-4f3f-b513-501ec601ec40-serviceca\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-cni-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-env-overrides\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysctl-conf\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-var-lib-kubelet\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.551990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-cnibin\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.552018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63cfb73d-a2d1-41a6-8911-03772f734c2d-kubelet-config\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.552038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-socket-dir-parent\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.552054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-kubelet\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.552076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-modprobe-d\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.552093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-multus-certs\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.552110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-run-netns\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.647630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.556274 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:57:30.648216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.560941 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 14:52:29 +0000 UTC" deadline="2028-02-14 20:56:00.041889053 +0000 UTC" Jul 19 14:57:30.648216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.560962 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13805h58m29.480929918s" Jul 19 14:57:30.648216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.581790 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-26plf" Jul 19 14:57:30.648216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.590344 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-26plf" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-systemd\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvqm\" (UniqueName: \"kubernetes.io/projected/88f133f4-9d7a-4622-8eb8-97c9a630fbde-kube-api-access-7dvqm\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-run\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-socket-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-cni-binary-copy\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-systemd\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-system-cni-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-system-cni-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovnkube-config\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-socket-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-run-ovn-kubernetes\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-registration-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-registration-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.815216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-run\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1e81ea0-1dea-466e-a980-176b3a996a7d-hosts-file\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-log-socket\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovn-node-metrics-cert\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zcjm\" (UniqueName: \"kubernetes.io/projected/36be97b5-0bc8-41ad-847e-e8b817a9cf46-kube-api-access-6zcjm\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-log-socket\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1e81ea0-1dea-466e-a980-176b3a996a7d-hosts-file\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/754bd0cf-54de-4f3f-b513-501ec601ec40-host\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.652977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rps8f\" (UniqueName: \"kubernetes.io/projected/9f37c79b-1c46-4400-81a3-c64d9a2384c3-kube-api-access-rps8f\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-conf-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/754bd0cf-54de-4f3f-b513-501ec601ec40-host\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.653040 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-node-log\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-conf-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-cni-bin\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.653121 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:57:31.153099793 +0000 UTC m=+2.017374581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:30.816475 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-cni-bin\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-host\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/266b3d70-24cc-469e-8906-d6822c6beaa7-konnectivity-ca\") pod \"konnectivity-agent-mx6jp\" (UID: \"266b3d70-24cc-469e-8906-d6822c6beaa7\") " pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-node-log\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/754bd0cf-54de-4f3f-b513-501ec601ec40-serviceca\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-cni-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-env-overrides\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653182 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-host\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysctl-conf\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-var-lib-kubelet\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-cnibin\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63cfb73d-a2d1-41a6-8911-03772f734c2d-kubelet-config\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-socket-dir-parent\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-kubelet\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-cni-dir\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-modprobe-d\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-multus-certs\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.817346 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-cnibin\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-run-netns\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-sys-fs\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vt27\" (UniqueName: \"kubernetes.io/projected/c34f641b-e476-47c0-9dce-506e0d8b2c36-kube-api-access-9vt27\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-var-lib-kubelet\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysctl-conf\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f37c79b-1c46-4400-81a3-c64d9a2384c3-iptables-alerter-script\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-k8s-cni-cncf-io\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-cni-bin\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-sys-fs\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-env-overrides\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-modprobe-d\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-systemd-units\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/266b3d70-24cc-469e-8906-d6822c6beaa7-konnectivity-ca\") pod \"konnectivity-agent-mx6jp\" (UID: \"266b3d70-24cc-469e-8906-d6822c6beaa7\") " pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-slash\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-systemd-units\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-multus-certs\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-run-netns\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-multus-socket-dir-parent\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-kubelet\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63cfb73d-a2d1-41a6-8911-03772f734c2d-kubelet-config\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.653820 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-slash\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-k8s-cni-cncf-io\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-cni-bin\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.653875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:57:31.153859289 +0000 UTC m=+2.018134083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/754bd0cf-54de-4f3f-b513-501ec601ec40-serviceca\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-cni-binary-copy\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b035e097-e89a-46e2-ac9e-83543a280851-cni-binary-copy\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-var-lib-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.653999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-cni-netd\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.818804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysctl-d\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgrs\" (UniqueName: \"kubernetes.io/projected/a4142486-9667-4b0c-9304-d6edc4c4d09b-kube-api-access-7kgrs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-device-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-os-release\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysctl-d\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-var-lib-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-cni-netd\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.654345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.656152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-os-release\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.660874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e81ea0-1dea-466e-a980-176b3a996a7d-tmp-dir\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.660910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-device-dir\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.660923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-kubernetes\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.660953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovnkube-config\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.660962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/266b3d70-24cc-469e-8906-d6822c6beaa7-agent-certs\") pod \"konnectivity-agent-mx6jp\" (UID: \"266b3d70-24cc-469e-8906-d6822c6beaa7\") " pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.660998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-kubernetes\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.819387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovnkube-script-lib\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63cfb73d-a2d1-41a6-8911-03772f734c2d-dbus\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f37c79b-1c46-4400-81a3-c64d9a2384c3-host-slash\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-os-release\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-kubelet\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-etc-kubernetes\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f37c79b-1c46-4400-81a3-c64d9a2384c3-host-slash\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysconfig\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e81ea0-1dea-466e-a980-176b3a996a7d-tmp-dir\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm8s\" (UniqueName: \"kubernetes.io/projected/754bd0cf-54de-4f3f-b513-501ec601ec40-kube-api-access-ghm8s\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-os-release\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-netns\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-cnibin\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-etc-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f37c79b-1c46-4400-81a3-c64d9a2384c3-iptables-alerter-script\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:30.820009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-tuned\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwh7\" (UniqueName: \"kubernetes.io/projected/ae881e3e-1525-4350-9db6-90a307e44924-kube-api-access-slwh7\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-cni-multus\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dc6h\" (UniqueName: \"kubernetes.io/projected/b035e097-e89a-46e2-ac9e-83543a280851-kube-api-access-8dc6h\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-ovn\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-systemd\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djhm\" (UniqueName: \"kubernetes.io/projected/c1e81ea0-1dea-466e-a980-176b3a996a7d-kube-api-access-7djhm\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-etc-kubernetes\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b035e097-e89a-46e2-ac9e-83543a280851-multus-daemon-config\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-sys\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-sysconfig\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63cfb73d-a2d1-41a6-8911-03772f734c2d-dbus\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-lib-modules\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36be97b5-0bc8-41ad-847e-e8b817a9cf46-tmp\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-system-cni-dir\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.820864 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-hostroot\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-hostroot\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.662065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-cni-multus\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.661563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-cnibin\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.662242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-run-ovn\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.662275 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-systemd\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.662608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-run-netns\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.662651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b035e097-e89a-46e2-ac9e-83543a280851-host-var-lib-kubelet\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.663311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b035e097-e89a-46e2-ac9e-83543a280851-multus-daemon-config\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.663392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-sys\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.663470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36be97b5-0bc8-41ad-847e-e8b817a9cf46-lib-modules\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.663834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae881e3e-1525-4350-9db6-90a307e44924-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/266b3d70-24cc-469e-8906-d6822c6beaa7-agent-certs\") pod \"konnectivity-agent-mx6jp\" (UID: \"266b3d70-24cc-469e-8906-d6822c6beaa7\") " pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36be97b5-0bc8-41ad-847e-e8b817a9cf46-etc-tuned\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664219 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovn-node-metrics-cert\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zcjm\" (UniqueName: \"kubernetes.io/projected/36be97b5-0bc8-41ad-847e-e8b817a9cf46-kube-api-access-6zcjm\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:30.821473 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b035e097-e89a-46e2-ac9e-83543a280851-cni-binary-copy\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f133f4-9d7a-4622-8eb8-97c9a630fbde-etc-openvswitch\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c34f641b-e476-47c0-9dce-506e0d8b2c36-system-cni-dir\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f133f4-9d7a-4622-8eb8-97c9a630fbde-ovnkube-script-lib\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgrs\" (UniqueName: \"kubernetes.io/projected/a4142486-9667-4b0c-9304-d6edc4c4d09b-kube-api-access-7kgrs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.664951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c34f641b-e476-47c0-9dce-506e0d8b2c36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.666286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rps8f\" (UniqueName: \"kubernetes.io/projected/9f37c79b-1c46-4400-81a3-c64d9a2384c3-kube-api-access-rps8f\") pod \"iptables-alerter-xqbcl\" (UID: \"9f37c79b-1c46-4400-81a3-c64d9a2384c3\") " pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.666451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36be97b5-0bc8-41ad-847e-e8b817a9cf46-tmp\") pod \"tuned-49djs\" (UID: \"36be97b5-0bc8-41ad-847e-e8b817a9cf46\") " pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.666797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvqm\" (UniqueName: \"kubernetes.io/projected/88f133f4-9d7a-4622-8eb8-97c9a630fbde-kube-api-access-7dvqm\") pod \"ovnkube-node-h4h72\" (UID: \"88f133f4-9d7a-4622-8eb8-97c9a630fbde\") " pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.666950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vt27\" (UniqueName: \"kubernetes.io/projected/c34f641b-e476-47c0-9dce-506e0d8b2c36-kube-api-access-9vt27\") pod \"multus-additional-cni-plugins-hcjp9\" (UID: \"c34f641b-e476-47c0-9dce-506e0d8b2c36\") " pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.668506 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.668522 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.668532 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:30.668592 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:57:31.168565239 +0000 UTC m=+2.032840014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.671616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dc6h\" (UniqueName: \"kubernetes.io/projected/b035e097-e89a-46e2-ac9e-83543a280851-kube-api-access-8dc6h\") pod \"multus-c689m\" (UID: \"b035e097-e89a-46e2-ac9e-83543a280851\") " pod="openshift-multus/multus-c689m" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.671765 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghm8s\" (UniqueName: \"kubernetes.io/projected/754bd0cf-54de-4f3f-b513-501ec601ec40-kube-api-access-ghm8s\") pod \"node-ca-mscg5\" (UID: \"754bd0cf-54de-4f3f-b513-501ec601ec40\") " pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.672005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djhm\" (UniqueName: \"kubernetes.io/projected/c1e81ea0-1dea-466e-a980-176b3a996a7d-kube-api-access-7djhm\") pod \"node-resolver-7k7rd\" (UID: \"c1e81ea0-1dea-466e-a980-176b3a996a7d\") " pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:31.010394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.675676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwh7\" (UniqueName: \"kubernetes.io/projected/ae881e3e-1525-4350-9db6-90a307e44924-kube-api-access-slwh7\") pod \"aws-ebs-csi-driver-node-4xh8w\" (UID: \"ae881e3e-1525-4350-9db6-90a307e44924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.683562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-49djs" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.705326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.732168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7k7rd" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.744075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mscg5" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.860323 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a6110787fb36561c0436ce6584028c.slice/crio-f426ab71fb232b0d6f47fca81c62026f526667c781e43fc8be7773ef132cbf30 WatchSource:0}: Error finding container f426ab71fb232b0d6f47fca81c62026f526667c781e43fc8be7773ef132cbf30: Status 404 returned error can't find the container with id f426ab71fb232b0d6f47fca81c62026f526667c781e43fc8be7773ef132cbf30 Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.860691 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef091f26c73031458c515489d9e0ec4.slice/crio-a3973c34cbe41439e230b14c97ccf921ae2843f1759a5e9b22c007db22f1bc1b WatchSource:0}: Error finding container a3973c34cbe41439e230b14c97ccf921ae2843f1759a5e9b22c007db22f1bc1b: Status 404 returned error can't find the container with id a3973c34cbe41439e230b14c97ccf921ae2843f1759a5e9b22c007db22f1bc1b Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.864497 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.870044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqbcl" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.879245 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f37c79b_1c46_4400_81a3_c64d9a2384c3.slice/crio-a8f2a9f59d95ce12153e200e3f4779ba22877f67f864acc3842225825f1716a4 WatchSource:0}: Error finding container a8f2a9f59d95ce12153e200e3f4779ba22877f67f864acc3842225825f1716a4: Status 404 returned error can't find the container with id a8f2a9f59d95ce12153e200e3f4779ba22877f67f864acc3842225825f1716a4 Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.891489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c689m" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.900365 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb035e097_e89a_46e2_ac9e_83543a280851.slice/crio-faeb6e8f4d20e543c2c17704381c756b0ccb83cecc53d457300d2b4414ea776e WatchSource:0}: Error finding container faeb6e8f4d20e543c2c17704381c756b0ccb83cecc53d457300d2b4414ea776e: Status 404 returned error can't find the container with id faeb6e8f4d20e543c2c17704381c756b0ccb83cecc53d457300d2b4414ea776e Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.913706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.914403 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.921908 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae881e3e_1525_4350_9db6_90a307e44924.slice/crio-ba02dc962bab5c1840517c94b6ebac811ae4c6aa868fa576f1cb659cdbc2fd1c WatchSource:0}: Error finding container ba02dc962bab5c1840517c94b6ebac811ae4c6aa868fa576f1cb659cdbc2fd1c: Status 404 returned error can't find the container with id ba02dc962bab5c1840517c94b6ebac811ae4c6aa868fa576f1cb659cdbc2fd1c Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.936650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.946348 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266b3d70_24cc_469e_8906_d6822c6beaa7.slice/crio-6e29932697b437d45fd78df00eb165bbed77010fd99fddb6f1b4b84728d930ff WatchSource:0}: Error finding container 6e29932697b437d45fd78df00eb165bbed77010fd99fddb6f1b4b84728d930ff: Status 404 returned error can't find the container with id 6e29932697b437d45fd78df00eb165bbed77010fd99fddb6f1b4b84728d930ff Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:30.953225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:31.012483 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:30.967196 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f133f4_9d7a_4622_8eb8_97c9a630fbde.slice/crio-92227dbf482414a390c72517667cc9ef3d93b0106649fff05066019c4efe18c2 WatchSource:0}: Error finding container 92227dbf482414a390c72517667cc9ef3d93b0106649fff05066019c4efe18c2: Status 404 returned error can't find the container with id 92227dbf482414a390c72517667cc9ef3d93b0106649fff05066019c4efe18c2 Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:31.068082 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754bd0cf_54de_4f3f_b513_501ec601ec40.slice/crio-3c5a844741ed4c91acf206a0aa35c1e6bc6d8516ae626873c2e3285eeea2e969 WatchSource:0}: Error finding container 3c5a844741ed4c91acf206a0aa35c1e6bc6d8516ae626873c2e3285eeea2e969: Status 404 returned error can't find the container with id 3c5a844741ed4c91acf206a0aa35c1e6bc6d8516ae626873c2e3285eeea2e969 Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:57:31.086612 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e81ea0_1dea_466e_a980_176b3a996a7d.slice/crio-ae823205ab130acd0883be46b1102f7132d14b7487e6ba6f72404255c81aa2a2 WatchSource:0}: Error finding container ae823205ab130acd0883be46b1102f7132d14b7487e6ba6f72404255c81aa2a2: Status 404 returned error can't find the container with id ae823205ab130acd0883be46b1102f7132d14b7487e6ba6f72404255c81aa2a2 Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.166567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.166650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.166725 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.166728 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.166773 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:57:32.166761383 +0000 UTC m=+3.031036158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:31.212274 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.166788 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:57:32.166780078 +0000 UTC m=+3.031054852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:31.424291 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.267840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:31.424291 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.267989 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:31.424291 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.268009 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:31.424291 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.268018 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:31.424291 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:31.268077 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:57:32.26805888 +0000 UTC m=+3.132333655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:31.579371 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.547913 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:31.594635 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.593311 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:52:30 +0000 UTC" deadline="2028-02-02 14:58:02.27120001 +0000 UTC" Jul 19 14:57:31.594635 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.593362 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13512h0m30.677840646s" Jul 19 14:57:31.751116 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.748930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqbcl" event={"ID":"9f37c79b-1c46-4400-81a3-c64d9a2384c3","Type":"ContainerStarted","Data":"a8f2a9f59d95ce12153e200e3f4779ba22877f67f864acc3842225825f1716a4"} Jul 19 14:57:31.754009 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.751951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" event={"ID":"b4a6110787fb36561c0436ce6584028c","Type":"ContainerStarted","Data":"f426ab71fb232b0d6f47fca81c62026f526667c781e43fc8be7773ef132cbf30"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.760604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mscg5" event={"ID":"754bd0cf-54de-4f3f-b513-501ec601ec40","Type":"ContainerStarted","Data":"3c5a844741ed4c91acf206a0aa35c1e6bc6d8516ae626873c2e3285eeea2e969"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.765138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-49djs" event={"ID":"36be97b5-0bc8-41ad-847e-e8b817a9cf46","Type":"ContainerStarted","Data":"9528225812e6a276b6a4c77cecd2ffaf97f69865a0c68e7810d6aebb37aada31"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.773841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mx6jp" event={"ID":"266b3d70-24cc-469e-8906-d6822c6beaa7","Type":"ContainerStarted","Data":"6e29932697b437d45fd78df00eb165bbed77010fd99fddb6f1b4b84728d930ff"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.779349 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" event={"ID":"4ef091f26c73031458c515489d9e0ec4","Type":"ContainerStarted","Data":"a3973c34cbe41439e230b14c97ccf921ae2843f1759a5e9b22c007db22f1bc1b"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.781899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7k7rd" event={"ID":"c1e81ea0-1dea-466e-a980-176b3a996a7d","Type":"ContainerStarted","Data":"ae823205ab130acd0883be46b1102f7132d14b7487e6ba6f72404255c81aa2a2"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.784817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerStarted","Data":"7f7ef289a68b1dee1d5cdf3b715aa1adfd40ee6d46742aa140a34ce0e21f6c7f"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.788331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"92227dbf482414a390c72517667cc9ef3d93b0106649fff05066019c4efe18c2"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.791013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" event={"ID":"ae881e3e-1525-4350-9db6-90a307e44924","Type":"ContainerStarted","Data":"ba02dc962bab5c1840517c94b6ebac811ae4c6aa868fa576f1cb659cdbc2fd1c"} Jul 19 14:57:31.819409 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:31.793444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c689m" event={"ID":"b035e097-e89a-46e2-ac9e-83543a280851","Type":"ContainerStarted","Data":"faeb6e8f4d20e543c2c17704381c756b0ccb83cecc53d457300d2b4414ea776e"} Jul 19 14:57:32.184691 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.184605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:32.184849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.184705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:32.184849 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.184772 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:32.184849 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.184812 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:32.184849 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.184848 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:57:34.184826325 +0000 UTC m=+5.049101102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:32.185064 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.184869 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:57:34.184858444 +0000 UTC m=+5.049133225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:32.286383 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.285807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:32.286383 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.285981 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:32.286383 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.286002 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:32.286383 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.286014 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:32.286383 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.286070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:57:34.286053054 +0000 UTC m=+5.150327842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:32.594572 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.594426 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:52:30 +0000 UTC" deadline="2028-02-13 09:17:16.135456646 +0000 UTC" Jul 19 14:57:32.594572 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.594485 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13770h19m43.54097537s" Jul 19 14:57:32.657644 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.656747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:32.657644 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.656873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:32.657644 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.657232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:32.657644 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.657315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:32.657644 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.657379 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:32.657644 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:32.657463 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:32.860651 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:32.860219 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:34.202542 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:34.202504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:34.203011 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:34.202627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:34.203011 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.202664 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:34.203011 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.202757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:57:38.202726956 +0000 UTC m=+9.067001731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:34.203011 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.202781 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:34.203011 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.202834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:57:38.202818343 +0000 UTC m=+9.067093121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:34.303749 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:34.303702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:34.303933 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.303920 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:34.303992 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.303945 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:34.303992 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.303960 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:34.304101 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.304020 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:57:38.304000737 +0000 UTC m=+9.168275515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:34.656461 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:34.656245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:34.656461 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.656372 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:34.656702 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:34.656460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:34.656702 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.656572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:34.656702 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:34.656662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:34.656851 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:34.656755 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:36.656732 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:36.656639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:36.657197 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:36.656768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:36.657197 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:36.656884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:36.657197 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:36.657015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:36.657197 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:36.657086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:36.657197 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:36.657169 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:38.239068 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:38.239027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:38.239488 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:38.239106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:38.239488 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.239173 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:38.239488 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.239254 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:57:46.239233564 +0000 UTC m=+17.103508338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:38.239488 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.239262 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:38.239488 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.239317 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:57:46.239299345 +0000 UTC m=+17.103574134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:38.340322 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:38.339682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:38.340322 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.339901 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:38.340322 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.339919 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:38.340322 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.339932 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:38.340322 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.339991 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:57:46.339971951 +0000 UTC m=+17.204246729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:38.656375 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:38.656248 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:38.656573 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.656406 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:38.656573 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:38.656541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:38.656745 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.656665 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:38.656745 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:38.656715 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:38.656854 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:38.656787 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:40.656819 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:40.656783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:40.657332 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:40.656926 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:40.657332 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:40.657066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:40.657332 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:40.657153 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:40.657332 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:40.657209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:40.657332 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:40.657271 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:42.656591 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:42.656546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:42.656591 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:42.656569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:42.657013 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:42.656563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:42.657013 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:42.656687 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:42.657013 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:42.656763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:42.657013 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:42.656866 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:44.656551 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:44.656509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:44.656551 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:44.656535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:44.657045 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:44.656539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:44.657045 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:44.656671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:44.657045 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:44.656764 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:44.657045 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:44.656854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:46.302210 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:46.302173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:46.302669 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:46.302229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:46.302669 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.302334 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:46.302669 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.302340 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:46.302669 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.302396 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:58:02.302376052 +0000 UTC m=+33.166650846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:46.302669 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.302418 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:02.302407082 +0000 UTC m=+33.166681857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:46.402794 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:46.402759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:46.402967 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.402937 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:46.402967 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.402964 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:46.403062 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.402976 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:46.403062 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.403030 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:02.403013924 +0000 UTC m=+33.267288700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:46.655964 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:46.655933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:46.655964 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:46.655958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:46.656158 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:46.655933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:46.656158 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.656033 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:46.656158 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.656112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:46.656256 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:46.656185 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:48.656495 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:48.656454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:48.656973 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:48.656454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:48.656973 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:48.656603 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:48.656973 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:48.656473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:48.656973 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:48.656763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:48.656973 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:48.656810 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:50.656977 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:50.656794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:50.657364 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:50.657036 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:50.657364 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:50.656794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:50.657364 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:50.656804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:50.657364 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:50.657170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:50.657364 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:50.657234 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:51.848938 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.848905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-49djs" event={"ID":"36be97b5-0bc8-41ad-847e-e8b817a9cf46","Type":"ContainerStarted","Data":"3508761a44bb3ab4b7f8a54add1ba3ff78922607d0f51b03c9de5b2927285ddf"} Jul 19 14:57:51.852216 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.852194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"2484c9a2f5b7fe56b62bb28e4817ac22a4c57878ac59f201f1d3f2d44794b3c3"} Jul 19 14:57:51.852328 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.852267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"34d512585bf4561c3f5b308bfa61e101bf2d5edfc58f6aa295e8f7663fb0ff41"} Jul 19 14:57:51.852328 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.852318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"f6be1c4c7e8ec7850910991613e7c948e262d8210f36e95c76b1b1520ad545c3"} Jul 19 14:57:51.853566 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.853545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c689m" event={"ID":"b035e097-e89a-46e2-ac9e-83543a280851","Type":"ContainerStarted","Data":"4bc2fdb343ba4991f866256a840f3738733e1f55755875e544ff2071a663dc11"} Jul 19 14:57:51.854775 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.854754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" event={"ID":"b4a6110787fb36561c0436ce6584028c","Type":"ContainerStarted","Data":"f07073c24d72326816df26bca6aed67eda115f08998ee61744d1a9b7319f29ff"} Jul 19 14:57:51.874977 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.874931 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-49djs" podStartSLOduration=1.6234132319999999 podStartE2EDuration="21.874919s" totalImagesPullingTime="20.251505768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:51.874681014 +0000 UTC m=+22.738955811" watchObservedRunningTime="2026-07-19 14:57:51.874919 +0000 UTC m=+22.739193796" Jul 19 14:57:51.888270 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.888233 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" podStartSLOduration=21.888218888 podStartE2EDuration="21.888218888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:51.887688303 +0000 UTC m=+22.751963101" watchObservedRunningTime="2026-07-19 14:57:51.888218888 +0000 UTC m=+22.752493685" Jul 19 14:57:51.912422 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:51.912381 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c689m" podStartSLOduration=2.533690274 podStartE2EDuration="22.912365586s" totalImagesPullingTime="20.378675312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:51.912316424 +0000 UTC m=+22.776591226" watchObservedRunningTime="2026-07-19 14:57:51.912365586 +0000 UTC m=+22.776640383" Jul 19 14:57:52.656733 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.656495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:52.656891 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.656495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:52.656891 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:52.656777 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:52.656891 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.656495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:52.656986 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:52.656841 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:52.656986 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:52.656909 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:52.858385 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.858350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mscg5" event={"ID":"754bd0cf-54de-4f3f-b513-501ec601ec40","Type":"ContainerStarted","Data":"e9d63a5cf00c3ee6d07321e7fa5f4e26bf2517442f358ae9f6dfac1046c7fa19"} Jul 19 14:57:52.859839 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.859817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mx6jp" event={"ID":"266b3d70-24cc-469e-8906-d6822c6beaa7","Type":"ContainerStarted","Data":"38b6a7c6d2729be54a73beb178233b1428c3ad163418d166f65cb7e511efb473"} Jul 19 14:57:52.861239 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.861216 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ef091f26c73031458c515489d9e0ec4" containerID="287f194859cc209e9a614cb2fcd9a0fff60807bbe9dd13ac0371db884e010cce" exitCode=0 Jul 19 14:57:52.861331 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.861291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" event={"ID":"4ef091f26c73031458c515489d9e0ec4","Type":"ContainerDied","Data":"287f194859cc209e9a614cb2fcd9a0fff60807bbe9dd13ac0371db884e010cce"} Jul 19 14:57:52.865821 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.865790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7k7rd" event={"ID":"c1e81ea0-1dea-466e-a980-176b3a996a7d","Type":"ContainerStarted","Data":"bea32636381df4b5ec4cc5b4347fdea14cf048e0515b3bdc2ec107f2d394f743"} Jul 19 14:57:52.867275 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.867255 2580 generic.go:356] "Generic (PLEG): container finished" podID="c34f641b-e476-47c0-9dce-506e0d8b2c36" containerID="4f6f35d817328bb8f7026d3e8030fb5a45c80edfdbd80a451b54abef143420bb" exitCode=0 Jul 19 14:57:52.867364 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.867333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerDied","Data":"4f6f35d817328bb8f7026d3e8030fb5a45c80edfdbd80a451b54abef143420bb"} Jul 19 14:57:52.870118 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.870046 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 14:57:52.870403 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.870385 2580 generic.go:356] "Generic (PLEG): container finished" podID="88f133f4-9d7a-4622-8eb8-97c9a630fbde" containerID="34d512585bf4561c3f5b308bfa61e101bf2d5edfc58f6aa295e8f7663fb0ff41" exitCode=1 Jul 19 14:57:52.870478 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.870444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerDied","Data":"34d512585bf4561c3f5b308bfa61e101bf2d5edfc58f6aa295e8f7663fb0ff41"} Jul 19 14:57:52.870533 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.870501 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"2efa9a5f64f30c589e2d4785b7eaefdb64394e7ba04a74319ab6ed91611163c3"} Jul 19 14:57:52.870574 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.870537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"c5ad3a1e239f514777a5bc3112f9085c9e89f23b8fb3a4d7026a1fd89910f83b"} Jul 19 14:57:52.870574 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.870566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"a602d73144fdbd85438be2ebeb5cdec7caa9c463b6840dbec92854b32d544021"} Jul 19 14:57:52.871733 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.871705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" event={"ID":"ae881e3e-1525-4350-9db6-90a307e44924","Type":"ContainerStarted","Data":"6245fbc57c375163bcc2a4b53ba98cc0eb247d4779641b79d773563362ca4621"} Jul 19 14:57:52.873028 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.873003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqbcl" event={"ID":"9f37c79b-1c46-4400-81a3-c64d9a2384c3","Type":"ContainerStarted","Data":"9eb958039682335e5af0e960d5fa4f0f3089e512cce4634750cb3c3f93f9c551"} Jul 19 14:57:52.875172 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.875138 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mscg5" podStartSLOduration=2.674236948 podStartE2EDuration="22.875127589s" totalImagesPullingTime="20.200890641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:52.875042345 +0000 UTC m=+23.739317142" watchObservedRunningTime="2026-07-19 14:57:52.875127589 +0000 UTC m=+23.739402390" Jul 19 14:57:52.905152 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.904654 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xqbcl" podStartSLOduration=3.509770891 podStartE2EDuration="23.904637211s" totalImagesPullingTime="20.39486632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:52.90443523 +0000 UTC m=+23.768710026" watchObservedRunningTime="2026-07-19 14:57:52.904637211 +0000 UTC m=+23.768912008" Jul 19 14:57:52.918874 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.918756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mx6jp" podStartSLOduration=3.620257928 podStartE2EDuration="23.918737956s" totalImagesPullingTime="20.298480028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:52.918235098 +0000 UTC m=+23.782509892" watchObservedRunningTime="2026-07-19 14:57:52.918737956 +0000 UTC m=+23.783012755" Jul 19 14:57:52.932163 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:52.932127 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7k7rd" podStartSLOduration=2.74455306 podStartE2EDuration="22.932116326s" totalImagesPullingTime="20.187563266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:52.931721389 +0000 UTC m=+23.795996186" watchObservedRunningTime="2026-07-19 14:57:52.932116326 +0000 UTC m=+23.796391122" Jul 19 14:57:53.880070 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:53.880033 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" event={"ID":"4ef091f26c73031458c515489d9e0ec4","Type":"ContainerStarted","Data":"00fc3f5ebd30055b78ff52c4de3a6e31c422b876525e46ef1ec09d1d0fabb641"} Jul 19 14:57:53.885698 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:53.885677 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 19 14:57:53.896618 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:53.896560 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" podStartSLOduration=23.89654502 podStartE2EDuration="23.89654502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:53.896236033 +0000 UTC m=+24.760510831" watchObservedRunningTime="2026-07-19 14:57:53.89654502 +0000 UTC m=+24.760819813" Jul 19 14:57:54.656799 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.656771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:54.656998 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.656772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:54.656998 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:54.656866 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:54.656998 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.656771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:54.656998 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:54.656955 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:54.657149 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:54.657070 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:54.662344 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.662269 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T14:57:53.885693743Z","UUID":"d26286d0-a1de-4000-a31e-4dccacf664c5","Handler":null,"Name":"","Endpoint":""} Jul 19 14:57:54.663623 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.663607 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 19 14:57:54.663719 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.663640 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 19 14:57:54.886014 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.885984 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 14:57:54.886505 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.886421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"5a67fda8f44e1cc6796450efd2e3514d7cda5babe3442a0b2e0e61e929659f58"} Jul 19 14:57:54.888538 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:54.888509 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" event={"ID":"ae881e3e-1525-4350-9db6-90a307e44924","Type":"ContainerStarted","Data":"fba4550bd7cb69624cddf419f1fa039cd72bcf6f1679d5453c2b4f85a90ce108"} Jul 19 14:57:55.892841 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:55.892797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" event={"ID":"ae881e3e-1525-4350-9db6-90a307e44924","Type":"ContainerStarted","Data":"78af04b7a75acad1abc779d6304a58d70a27e92a40693169f1946fb888346f46"} Jul 19 14:57:55.909071 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:55.909016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xh8w" podStartSLOduration=4.160939589 podStartE2EDuration="26.908994939s" totalImagesPullingTime="22.74805535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:55.908940569 +0000 UTC m=+26.773215391" watchObservedRunningTime="2026-07-19 14:57:55.908994939 +0000 UTC m=+26.773269726" Jul 19 14:57:56.656763 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:56.656553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:56.656927 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:56.656553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:56.656927 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:56.656869 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:56.656927 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:56.656553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:56.657151 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:56.656935 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:56.657151 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:56.657027 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:56.898858 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:56.898779 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 14:57:56.899550 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:56.899265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"0bece1e8bbdbc5f30a9b788875e586319c469719e9cf262b3f06bf7bcd27b8cf"} Jul 19 14:57:56.899964 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:56.899939 2580 scope.go:117] "RemoveContainer" containerID="34d512585bf4561c3f5b308bfa61e101bf2d5edfc58f6aa295e8f7663fb0ff41" Jul 19 14:57:57.377349 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:57.377315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:57.378103 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:57.378076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:57.901761 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:57.901726 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:57.902618 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:57.902573 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mx6jp" Jul 19 14:57:58.616176 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.615998 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rgb9j"] Jul 19 14:57:58.616347 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.616274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:58.616411 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:58.616388 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:57:58.619303 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.619270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xpbn5"] Jul 19 14:57:58.619427 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.619383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:57:58.619499 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:58.619468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:57:58.619950 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.619926 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gsvrk"] Jul 19 14:57:58.620192 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.620023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:57:58.620192 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:58.620105 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:57:58.905204 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.905174 2580 generic.go:356] "Generic (PLEG): container finished" podID="c34f641b-e476-47c0-9dce-506e0d8b2c36" containerID="89f8bb3bfcf3a09ad354f45c2f77df832a67f70289a0ab2f22f114ffdde39894" exitCode=0 Jul 19 14:57:58.905877 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.905249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerDied","Data":"89f8bb3bfcf3a09ad354f45c2f77df832a67f70289a0ab2f22f114ffdde39894"} Jul 19 14:57:58.909267 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.909247 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 14:57:58.909647 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.909616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" event={"ID":"88f133f4-9d7a-4622-8eb8-97c9a630fbde","Type":"ContainerStarted","Data":"9dd59d4669627c68575999847a96b822be950359f2c32578937dda84f85f11e4"} Jul 19 14:57:58.909942 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.909916 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:58.910057 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.909988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:58.910057 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.910039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:58.925259 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.925238 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:58.925365 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.925332 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:57:58.964036 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:58.963980 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" podStartSLOduration=9.432599875 podStartE2EDuration="29.963965697s" totalImagesPullingTime="20.531365822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:57:58.963919621 +0000 UTC m=+29.828194418" watchObservedRunningTime="2026-07-19 14:57:58.963965697 +0000 UTC m=+29.828240471" Jul 19 14:57:59.657077 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:57:59.657028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:57:59.657272 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:57:59.657148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:58:00.655982 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:00.655927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:00.655982 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:00.655971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:00.656384 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:00.656085 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:58:00.656384 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:00.656199 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:58:00.915605 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:00.915556 2580 generic.go:356] "Generic (PLEG): container finished" podID="c34f641b-e476-47c0-9dce-506e0d8b2c36" containerID="880894fea028ff3f24a15298ef244674359c7f6ea5402b30bc50712dd600625e" exitCode=0 Jul 19 14:58:00.915744 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:00.915624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerDied","Data":"880894fea028ff3f24a15298ef244674359c7f6ea5402b30bc50712dd600625e"} Jul 19 14:58:01.656620 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:01.656570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:01.657062 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:01.656732 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:58:02.321460 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.321427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:02.321635 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.321467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:02.321635 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.321562 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:02.321635 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.321569 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:02.321635 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.321630 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret podName:63cfb73d-a2d1-41a6-8911-03772f734c2d nodeName:}" failed. No retries permitted until 2026-07-19 14:58:34.321617662 +0000 UTC m=+65.185892437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret") pod "global-pull-secret-syncer-rgb9j" (UID: "63cfb73d-a2d1-41a6-8911-03772f734c2d") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:02.321788 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.321671 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs podName:a4142486-9667-4b0c-9304-d6edc4c4d09b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:34.321652447 +0000 UTC m=+65.185927228 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs") pod "network-metrics-daemon-xpbn5" (UID: "a4142486-9667-4b0c-9304-d6edc4c4d09b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:02.421810 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.421782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:02.421936 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.421906 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:02.421936 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.421921 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:02.421936 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.421929 2580 projected.go:196] Error preparing data for projected volume kube-api-access-q25xh for pod openshift-network-diagnostics/network-check-target-gsvrk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:02.422056 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.421976 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh podName:e0b8549c-1b3d-423f-9df2-a5a583a7194c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:34.421963223 +0000 UTC m=+65.286237999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q25xh" (UniqueName: "kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh") pod "network-check-target-gsvrk" (UID: "e0b8549c-1b3d-423f-9df2-a5a583a7194c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:02.656593 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.656551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:02.656745 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.656684 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpbn5" podUID="a4142486-9667-4b0c-9304-d6edc4c4d09b" Jul 19 14:58:02.656745 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.656730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:02.657016 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:02.656808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gsvrk" podUID="e0b8549c-1b3d-423f-9df2-a5a583a7194c" Jul 19 14:58:02.922244 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.922211 2580 generic.go:356] "Generic (PLEG): container finished" podID="c34f641b-e476-47c0-9dce-506e0d8b2c36" containerID="4d81be457643c82b2c657224d6313f665126a0364ca62d33724f9555d6e8ec3d" exitCode=0 Jul 19 14:58:02.922413 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:02.922264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerDied","Data":"4d81be457643c82b2c657224d6313f665126a0364ca62d33724f9555d6e8ec3d"} Jul 19 14:58:03.655915 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:03.655880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:03.656098 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:03.656010 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgb9j" podUID="63cfb73d-a2d1-41a6-8911-03772f734c2d" Jul 19 14:58:04.374548 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.374523 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeReady" Jul 19 14:58:04.375024 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.374701 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 19 14:58:04.410259 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.410228 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58d9c8db76-blrq5"] Jul 19 14:58:04.413790 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.413772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.416045 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.416016 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 19 14:58:04.416592 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.416558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 19 14:58:04.417049 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.416801 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 19 14:58:04.417523 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.417504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-znqlg\"" Jul 19 14:58:04.425088 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.424434 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d9c8db76-blrq5"] Jul 19 14:58:04.425374 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.425347 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ttnr8"] Jul 19 14:58:04.426212 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.426189 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 19 14:58:04.429641 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.429619 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ndgfw"] Jul 19 14:58:04.429770 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.429753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.433002 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.432979 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 19 14:58:04.433002 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.432998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 19 14:58:04.433159 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.433015 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvk2r\"" Jul 19 14:58:04.433305 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.433285 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.435202 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.435184 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g45s6\"" Jul 19 14:58:04.435506 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.435484 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 19 14:58:04.435506 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.435491 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 19 14:58:04.435795 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.435601 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.437481 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.437371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ttnr8"] Jul 19 14:58:04.440501 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.440316 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ndgfw"] Jul 19 14:58:04.535130 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-certificates\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535130 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c64568-904b-478f-a9d8-a8b9151a0748-config-volume\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.535130 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.535130 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kkr\" (UniqueName: \"kubernetes.io/projected/b6c64568-904b-478f-a9d8-a8b9151a0748-kube-api-access-62kkr\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83275949-83ec-4fb7-9ca7-7dd6ebe2ce10-cert\") pod \"ingress-canary-ndgfw\" (UID: \"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10\") " pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-image-registry-private-configuration\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgq7\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-kube-api-access-rmgq7\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-ca-trust-extracted\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c64568-904b-478f-a9d8-a8b9151a0748-tmp-dir\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.535405 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-installation-pull-secrets\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535646 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-trusted-ca\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535646 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-bound-sa-token\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.535646 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.535494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slpq\" (UniqueName: \"kubernetes.io/projected/83275949-83ec-4fb7-9ca7-7dd6ebe2ce10-kube-api-access-9slpq\") pod \"ingress-canary-ndgfw\" (UID: \"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10\") " pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.555015 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.554916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:04.636249 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slpq\" (UniqueName: \"kubernetes.io/projected/83275949-83ec-4fb7-9ca7-7dd6ebe2ce10-kube-api-access-9slpq\") pod \"ingress-canary-ndgfw\" (UID: \"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10\") " pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-certificates\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c64568-904b-478f-a9d8-a8b9151a0748-config-volume\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kkr\" (UniqueName: \"kubernetes.io/projected/b6c64568-904b-478f-a9d8-a8b9151a0748-kube-api-access-62kkr\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83275949-83ec-4fb7-9ca7-7dd6ebe2ce10-cert\") pod \"ingress-canary-ndgfw\" (UID: \"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10\") " pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-image-registry-private-configuration\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:04.636446 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgq7\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-kube-api-access-rmgq7\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-ca-trust-extracted\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:04.636528 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls podName:b6c64568-904b-478f-a9d8-a8b9151a0748 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.136507053 +0000 UTC m=+36.000781828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls") pod "dns-default-ttnr8" (UID: "b6c64568-904b-478f-a9d8-a8b9151a0748") : secret "dns-default-metrics-tls" not found Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:04.636536 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:04.636556 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d9c8db76-blrq5: secret "image-registry-tls" not found Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c64568-904b-478f-a9d8-a8b9151a0748-tmp-dir\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:04.636630 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls podName:6c3bf732-ba93-48f8-afb6-01c2a60f5c33 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.136613008 +0000 UTC m=+36.000887797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls") pod "image-registry-58d9c8db76-blrq5" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33") : secret "image-registry-tls" not found Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-installation-pull-secrets\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-trusted-ca\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.636765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-bound-sa-token\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.637085 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.636993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c64568-904b-478f-a9d8-a8b9151a0748-tmp-dir\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.637085 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.637065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-ca-trust-extracted\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.637322 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.637292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6c64568-904b-478f-a9d8-a8b9151a0748-config-volume\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.637552 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.637472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-certificates\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.638453 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.638422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-trusted-ca\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.641517 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.641492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-image-registry-private-configuration\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.641625 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.641542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-installation-pull-secrets\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.641703 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.641633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83275949-83ec-4fb7-9ca7-7dd6ebe2ce10-cert\") pod \"ingress-canary-ndgfw\" (UID: \"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10\") " pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.643908 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.643754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:04.646065 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.646040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgq7\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-kube-api-access-rmgq7\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.646201 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.646178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kkr\" (UniqueName: \"kubernetes.io/projected/b6c64568-904b-478f-a9d8-a8b9151a0748-kube-api-access-62kkr\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:04.646387 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.646366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-bound-sa-token\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:04.646537 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.646519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slpq\" (UniqueName: \"kubernetes.io/projected/83275949-83ec-4fb7-9ca7-7dd6ebe2ce10-kube-api-access-9slpq\") pod \"ingress-canary-ndgfw\" (UID: \"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10\") " pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.656853 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.656827 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:04.656956 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.656859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:04.659090 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.659038 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:58:04.659194 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.659173 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:58:04.659406 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.659379 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7fqw\"" Jul 19 14:58:04.659490 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.659407 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.659490 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.659409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hc8w7\"" Jul 19 14:58:04.748978 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.748942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ndgfw" Jul 19 14:58:04.857187 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.857163 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r"] Jul 19 14:58:04.862646 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.862627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" Jul 19 14:58:04.865798 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.865605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5fshq\"" Jul 19 14:58:04.866464 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.866442 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst"] Jul 19 14:58:04.871781 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.871741 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69"] Jul 19 14:58:04.872692 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.871894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:04.875133 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.875111 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv"] Jul 19 14:58:04.875811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.875460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:04.875914 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.875881 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 19 14:58:04.876929 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.876472 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:58:04.877137 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.877116 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.879573 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.878420 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cktd2\"" Jul 19 14:58:04.879573 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.878928 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 19 14:58:04.881697 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.879998 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z2gd4\"" Jul 19 14:58:04.881697 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.880274 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 19 14:58:04.881697 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.880646 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 19 14:58:04.881697 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.881247 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 19 14:58:04.883541 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.883338 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.886568 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.886549 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv"] Jul 19 14:58:04.887483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.886999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:04.889910 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.889771 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 19 14:58:04.890066 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.890031 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tf9jj\"" Jul 19 14:58:04.891798 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.891774 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 19 14:58:04.892700 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.892667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r"] Jul 19 14:58:04.892805 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.892701 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69"] Jul 19 14:58:04.892805 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.892734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst"] Jul 19 14:58:04.892805 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.892751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv"] Jul 19 14:58:04.892805 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.892768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv"] Jul 19 14:58:04.892805 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.892794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:04.895747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.895262 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 19 14:58:04.895747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.895534 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q54d7\"" Jul 19 14:58:04.895747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.895566 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.895747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.895597 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:58:04.909691 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.909667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ndgfw"] Jul 19 14:58:04.913753 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:04.913723 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83275949_83ec_4fb7_9ca7_7dd6ebe2ce10.slice/crio-e2520d97a468f100786933eccc1fa93a3e420983935975674cd739be391b16eb WatchSource:0}: Error finding container e2520d97a468f100786933eccc1fa93a3e420983935975674cd739be391b16eb: Status 404 returned error can't find the container with id e2520d97a468f100786933eccc1fa93a3e420983935975674cd739be391b16eb Jul 19 14:58:04.928647 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.928619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ndgfw" event={"ID":"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10","Type":"ContainerStarted","Data":"e2520d97a468f100786933eccc1fa93a3e420983935975674cd739be391b16eb"} Jul 19 14:58:04.960535 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.960508 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl"] Jul 19 14:58:04.963983 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.963963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" Jul 19 14:58:04.966339 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.966314 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-f662r"] Jul 19 14:58:04.966914 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.966868 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:58:04.967687 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.967666 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d2gx8\"" Jul 19 14:58:04.969817 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.969798 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75784d96f4-xmj78"] Jul 19 14:58:04.969965 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.969939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:04.973508 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.973492 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sb2j7"] Jul 19 14:58:04.973765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.973749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:04.976820 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.976801 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s"] Jul 19 14:58:04.976932 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.976894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:04.980067 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.980049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:04.987619 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.987598 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.988500 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.988479 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl"] Jul 19 14:58:04.993093 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.993074 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 19 14:58:04.993234 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.993216 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 19 14:58:04.994566 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.994551 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6j9dh\"" Jul 19 14:58:04.994672 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.994569 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 19 14:58:04.999414 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:04.999395 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s"] Jul 19 14:58:05.001594 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.001563 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 19 14:58:05.001676 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.001625 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 19 14:58:05.001747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.001677 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 19 14:58:05.001941 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.001924 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 19 14:58:05.002001 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.001978 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tvzzw\"" Jul 19 14:58:05.004712 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.004692 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 19 14:58:05.004712 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.004704 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 19 14:58:05.005983 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.005965 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 19 14:58:05.005983 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.005977 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:58:05.009277 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.009260 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 19 14:58:05.009373 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.009314 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-khlmm\"" Jul 19 14:58:05.009746 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.009570 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 19 14:58:05.009746 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.009634 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 19 14:58:05.010145 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.009972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 19 14:58:05.010145 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.010062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r7f6z\"" Jul 19 14:58:05.010979 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.010961 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 19 14:58:05.011452 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.011166 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 19 14:58:05.011794 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.011778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-f662r"] Jul 19 14:58:05.012954 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.012935 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75784d96f4-xmj78"] Jul 19 14:58:05.016722 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.016702 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:58:05.017934 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.017905 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 19 14:58:05.029338 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.029304 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sb2j7"] Jul 19 14:58:05.039897 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.039866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08f073a5-104f-4805-af4f-afd33d9c109b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.040027 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.039951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7047527-b902-4be6-94d5-9a3b377122d1-serving-cert\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.040027 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.039985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7047527-b902-4be6-94d5-9a3b377122d1-config\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.040027 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnztj\" (UniqueName: \"kubernetes.io/projected/b7047527-b902-4be6-94d5-9a3b377122d1-kube-api-access-jnztj\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.040160 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.040160 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9lm6\" (UniqueName: \"kubernetes.io/projected/cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238-kube-api-access-g9lm6\") pod \"network-check-source-7bc5b6bf85-m2c5r\" (UID: \"cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" Jul 19 14:58:05.040160 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/508b0e64-758e-4cc0-aa21-eb791dcc95f8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:05.040269 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:05.040269 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962z9\" (UniqueName: \"kubernetes.io/projected/08f073a5-104f-4805-af4f-afd33d9c109b-kube-api-access-962z9\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.040364 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmc5\" (UniqueName: \"kubernetes.io/projected/f426dcde-7439-4a0c-8800-f69e4833aa74-kube-api-access-8tmc5\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:05.040364 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.040292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:05.044001 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.043981 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 19 14:58:05.140497 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:05.140817 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zp9g\" (UniqueName: \"kubernetes.io/projected/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-kube-api-access-9zp9g\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.140919 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08f073a5-104f-4805-af4f-afd33d9c109b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.140919 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7047527-b902-4be6-94d5-9a3b377122d1-config\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.141030 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnztj\" (UniqueName: \"kubernetes.io/projected/b7047527-b902-4be6-94d5-9a3b377122d1-kube-api-access-jnztj\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.141030 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnj2\" (UniqueName: \"kubernetes.io/projected/8fbbbb06-6826-4239-9aea-1909f0723059-kube-api-access-cxnj2\") pod \"volume-data-source-validator-78cf4c6f6-k7rbl\" (UID: \"8fbbbb06-6826-4239-9aea-1909f0723059\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" Jul 19 14:58:05.141030 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.140991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.141172 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:05.141172 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-service-ca-bundle\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.141172 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmc5\" (UniqueName: \"kubernetes.io/projected/f426dcde-7439-4a0c-8800-f69e4833aa74-kube-api-access-8tmc5\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:05.141172 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.141368 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.141368 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-config\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.141368 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.141322 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:05.141368 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/125a2380-9e33-47d6-926a-9af477e93234-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.141368 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.141360 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d9c8db76-blrq5: secret "image-registry-tls" not found Jul 19 14:58:05.141646 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.141422 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls podName:6c3bf732-ba93-48f8-afb6-01c2a60f5c33 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.141408344 +0000 UTC m=+37.005683141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls") pod "image-registry-58d9c8db76-blrq5" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33") : secret "image-registry-tls" not found Jul 19 14:58:05.141646 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jft5\" (UniqueName: \"kubernetes.io/projected/17ac597b-4a6e-4a0f-a36e-5914098f469b-kube-api-access-5jft5\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.141646 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/125a2380-9e33-47d6-926a-9af477e93234-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.141646 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.141638 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7047527-b902-4be6-94d5-9a3b377122d1-config\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/08f073a5-104f-4805-af4f-afd33d9c109b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-tmp\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.141711 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls podName:08f073a5-104f-4805-af4f-afd33d9c109b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.641694078 +0000 UTC m=+36.505968858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-skx69" (UID: "08f073a5-104f-4805-af4f-afd33d9c109b") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mpq\" (UniqueName: \"kubernetes.io/projected/125a2380-9e33-47d6-926a-9af477e93234-kube-api-access-m8mpq\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-default-certificate\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.141824 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-stats-auth\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-snapshots\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7047527-b902-4be6-94d5-9a3b377122d1-serving-cert\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.141990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9lm6\" (UniqueName: \"kubernetes.io/projected/cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238-kube-api-access-g9lm6\") pod \"network-check-source-7bc5b6bf85-m2c5r\" (UID: \"cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/508b0e64-758e-4cc0-aa21-eb791dcc95f8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.142030 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-serving-cert\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.142122 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.142083 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls podName:b6c64568-904b-478f-a9d8-a8b9151a0748 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.142068893 +0000 UTC m=+37.006343668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls") pod "dns-default-ttnr8" (UID: "b6c64568-904b-478f-a9d8-a8b9151a0748") : secret "dns-default-metrics-tls" not found Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.142143 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962z9\" (UniqueName: \"kubernetes.io/projected/08f073a5-104f-4805-af4f-afd33d9c109b-kube-api-access-962z9\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vgd\" (UniqueName: \"kubernetes.io/projected/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-kube-api-access-78vgd\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.142195 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert podName:508b0e64-758e-4cc0-aa21-eb791dcc95f8 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.642181146 +0000 UTC m=+36.506456123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jwjbv" (UID: "508b0e64-758e-4cc0-aa21-eb791dcc95f8") : secret "networking-console-plugin-cert" not found Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-serving-cert\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-trusted-ca\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.142504 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.142470 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 14:58:05.142944 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.142538 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls podName:f426dcde-7439-4a0c-8800-f69e4833aa74 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.64252158 +0000 UTC m=+36.506796362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b46dv" (UID: "f426dcde-7439-4a0c-8800-f69e4833aa74") : secret "samples-operator-tls" not found Jul 19 14:58:05.142944 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.142739 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/508b0e64-758e-4cc0-aa21-eb791dcc95f8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:05.145121 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.145081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7047527-b902-4be6-94d5-9a3b377122d1-serving-cert\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.151814 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.151791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnztj\" (UniqueName: \"kubernetes.io/projected/b7047527-b902-4be6-94d5-9a3b377122d1-kube-api-access-jnztj\") pod \"service-ca-operator-5bcbc94c78-l6pst\" (UID: \"b7047527-b902-4be6-94d5-9a3b377122d1\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.152851 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.152831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmc5\" (UniqueName: \"kubernetes.io/projected/f426dcde-7439-4a0c-8800-f69e4833aa74-kube-api-access-8tmc5\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:05.160523 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.160504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962z9\" (UniqueName: \"kubernetes.io/projected/08f073a5-104f-4805-af4f-afd33d9c109b-kube-api-access-962z9\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.160641 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.160523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9lm6\" (UniqueName: \"kubernetes.io/projected/cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238-kube-api-access-g9lm6\") pod \"network-check-source-7bc5b6bf85-m2c5r\" (UID: \"cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" Jul 19 14:58:05.177465 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.177446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" Jul 19 14:58:05.188761 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.188735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" Jul 19 14:58:05.243682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.242844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/125a2380-9e33-47d6-926a-9af477e93234-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.243682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.242897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-tmp\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.243682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.242927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mpq\" (UniqueName: \"kubernetes.io/projected/125a2380-9e33-47d6-926a-9af477e93234-kube-api-access-m8mpq\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.243682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.242955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-default-certificate\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.243682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.243492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-tmp\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.242981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-stats-auth\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-snapshots\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-serving-cert\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vgd\" (UniqueName: \"kubernetes.io/projected/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-kube-api-access-78vgd\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-serving-cert\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-trusted-ca\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp9g\" (UniqueName: \"kubernetes.io/projected/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-kube-api-access-9zp9g\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnj2\" (UniqueName: \"kubernetes.io/projected/8fbbbb06-6826-4239-9aea-1909f0723059-kube-api-access-cxnj2\") pod \"volume-data-source-validator-78cf4c6f6-k7rbl\" (UID: \"8fbbbb06-6826-4239-9aea-1909f0723059\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-service-ca-bundle\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.244983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-config\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.245048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/125a2380-9e33-47d6-926a-9af477e93234-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.246398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.245093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jft5\" (UniqueName: \"kubernetes.io/projected/17ac597b-4a6e-4a0f-a36e-5914098f469b-kube-api-access-5jft5\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.246040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-trusted-ca\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.246366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-service-ca-bundle\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.246903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-snapshots\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.246958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-default-certificate\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.247106 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.7470866 +0000 UTC m=+36.611361388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : configmap references non-existent config key: service-ca.crt Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.248140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/125a2380-9e33-47d6-926a-9af477e93234-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.248729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/125a2380-9e33-47d6-926a-9af477e93234-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.248874 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.248933 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:05.748916595 +0000 UTC m=+36.613191382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : secret "router-metrics-certs-default" not found Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.249453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.251001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-config\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.251055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-stats-auth\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.251536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-serving-cert\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.253357 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.252902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-serving-cert\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.260430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.260387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp9g\" (UniqueName: \"kubernetes.io/projected/f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9-kube-api-access-9zp9g\") pod \"console-operator-565b7b569d-f662r\" (UID: \"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9\") " pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.260519 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.260468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnj2\" (UniqueName: \"kubernetes.io/projected/8fbbbb06-6826-4239-9aea-1909f0723059-kube-api-access-cxnj2\") pod \"volume-data-source-validator-78cf4c6f6-k7rbl\" (UID: \"8fbbbb06-6826-4239-9aea-1909f0723059\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" Jul 19 14:58:05.261951 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.261906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vgd\" (UniqueName: \"kubernetes.io/projected/bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e-kube-api-access-78vgd\") pod \"insights-operator-555d9d7885-sb2j7\" (UID: \"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e\") " pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.262680 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.262537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mpq\" (UniqueName: \"kubernetes.io/projected/125a2380-9e33-47d6-926a-9af477e93234-kube-api-access-m8mpq\") pod \"kube-storage-version-migrator-operator-7658d9b749-b864s\" (UID: \"125a2380-9e33-47d6-926a-9af477e93234\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.264295 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.264274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jft5\" (UniqueName: \"kubernetes.io/projected/17ac597b-4a6e-4a0f-a36e-5914098f469b-kube-api-access-5jft5\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.276911 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.276886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" Jul 19 14:58:05.284440 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.284417 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:05.300794 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.300686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" Jul 19 14:58:05.307197 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.307131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" Jul 19 14:58:05.339125 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.339089 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r"] Jul 19 14:58:05.342142 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.341714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst"] Jul 19 14:58:05.350681 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:05.350179 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7047527_b902_4be6_94d5_9a3b377122d1.slice/crio-639d36ca75b51042c2df75368e1e1368e2695cacde871504172a8117f88237fe WatchSource:0}: Error finding container 639d36ca75b51042c2df75368e1e1368e2695cacde871504172a8117f88237fe: Status 404 returned error can't find the container with id 639d36ca75b51042c2df75368e1e1368e2695cacde871504172a8117f88237fe Jul 19 14:58:05.482129 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.481799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-f662r"] Jul 19 14:58:05.486422 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:05.486386 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03f5080_5b9d_411a_b5f4_aab8f1f5e5c9.slice/crio-32d8efc4559ae6e2fa844a235479184288c48f0f897a76d05c30d4dde6e4677e WatchSource:0}: Error finding container 32d8efc4559ae6e2fa844a235479184288c48f0f897a76d05c30d4dde6e4677e: Status 404 returned error can't find the container with id 32d8efc4559ae6e2fa844a235479184288c48f0f897a76d05c30d4dde6e4677e Jul 19 14:58:05.494115 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.493787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl"] Jul 19 14:58:05.499025 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:05.498997 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbbbb06_6826_4239_9aea_1909f0723059.slice/crio-b7ec7b0f5124485dfd82257e26f7445c8a7106ff98c5594d32cccd2bb0394107 WatchSource:0}: Error finding container b7ec7b0f5124485dfd82257e26f7445c8a7106ff98c5594d32cccd2bb0394107: Status 404 returned error can't find the container with id b7ec7b0f5124485dfd82257e26f7445c8a7106ff98c5594d32cccd2bb0394107 Jul 19 14:58:05.519813 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.519784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s"] Jul 19 14:58:05.523106 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:05.523071 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125a2380_9e33_47d6_926a_9af477e93234.slice/crio-453c8883ba67aeea7419b9e98edf03bc73cc901423c581c3c7658519d120228b WatchSource:0}: Error finding container 453c8883ba67aeea7419b9e98edf03bc73cc901423c581c3c7658519d120228b: Status 404 returned error can't find the container with id 453c8883ba67aeea7419b9e98edf03bc73cc901423c581c3c7658519d120228b Jul 19 14:58:05.525057 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.525032 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-sb2j7"] Jul 19 14:58:05.529381 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:05.529354 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdf2c22_83b3_4475_98b0_6d1dcdaf8a7e.slice/crio-c5defa3cfc3c4acba861c9345c85f5d4fd00bd20caedc6ef4c12bbed5ee35adf WatchSource:0}: Error finding container c5defa3cfc3c4acba861c9345c85f5d4fd00bd20caedc6ef4c12bbed5ee35adf: Status 404 returned error can't find the container with id c5defa3cfc3c4acba861c9345c85f5d4fd00bd20caedc6ef4c12bbed5ee35adf Jul 19 14:58:05.651913 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.651878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:05.652061 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.651923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:05.652061 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.651975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:05.652061 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.652027 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:58:05.652237 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.652086 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 14:58:05.652237 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.652106 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:05.652237 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.652109 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert podName:508b0e64-758e-4cc0-aa21-eb791dcc95f8 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.652088266 +0000 UTC m=+37.516363056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jwjbv" (UID: "508b0e64-758e-4cc0-aa21-eb791dcc95f8") : secret "networking-console-plugin-cert" not found Jul 19 14:58:05.652237 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.652156 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls podName:f426dcde-7439-4a0c-8800-f69e4833aa74 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.652143353 +0000 UTC m=+37.516418128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b46dv" (UID: "f426dcde-7439-4a0c-8800-f69e4833aa74") : secret "samples-operator-tls" not found Jul 19 14:58:05.652237 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.652168 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls podName:08f073a5-104f-4805-af4f-afd33d9c109b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.652162138 +0000 UTC m=+37.516436913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-skx69" (UID: "08f073a5-104f-4805-af4f-afd33d9c109b") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:05.655856 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.655840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:05.657948 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.657928 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:58:05.753519 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.753484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.753519 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.753525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:05.753776 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.753713 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:58:05.753776 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.753717 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.75369333 +0000 UTC m=+37.617968105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : configmap references non-existent config key: service-ca.crt Jul 19 14:58:05.753776 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:05.753774 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:06.753758361 +0000 UTC m=+37.618033154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : secret "router-metrics-certs-default" not found Jul 19 14:58:05.933453 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.933358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-f662r" event={"ID":"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9","Type":"ContainerStarted","Data":"32d8efc4559ae6e2fa844a235479184288c48f0f897a76d05c30d4dde6e4677e"} Jul 19 14:58:05.934778 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.934748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" event={"ID":"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e","Type":"ContainerStarted","Data":"c5defa3cfc3c4acba861c9345c85f5d4fd00bd20caedc6ef4c12bbed5ee35adf"} Jul 19 14:58:05.936476 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.936427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" event={"ID":"125a2380-9e33-47d6-926a-9af477e93234","Type":"ContainerStarted","Data":"453c8883ba67aeea7419b9e98edf03bc73cc901423c581c3c7658519d120228b"} Jul 19 14:58:05.937777 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.937744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" event={"ID":"8fbbbb06-6826-4239-9aea-1909f0723059","Type":"ContainerStarted","Data":"b7ec7b0f5124485dfd82257e26f7445c8a7106ff98c5594d32cccd2bb0394107"} Jul 19 14:58:05.939166 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.939133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" event={"ID":"cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238","Type":"ContainerStarted","Data":"e609aa7292df92b45b9e838890084895b88d18e1e4234325685d060116f91cd6"} Jul 19 14:58:05.940429 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:05.940403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" event={"ID":"b7047527-b902-4be6-94d5-9a3b377122d1","Type":"ContainerStarted","Data":"639d36ca75b51042c2df75368e1e1368e2695cacde871504172a8117f88237fe"} Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.162158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.162255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.162431 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.162495 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls podName:b6c64568-904b-478f-a9d8-a8b9151a0748 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.162476195 +0000 UTC m=+39.026750972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls") pod "dns-default-ttnr8" (UID: "b6c64568-904b-478f-a9d8-a8b9151a0748") : secret "dns-default-metrics-tls" not found Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.162930 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.162946 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d9c8db76-blrq5: secret "image-registry-tls" not found Jul 19 14:58:06.163022 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.162994 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls podName:6c3bf732-ba93-48f8-afb6-01c2a60f5c33 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.16297815 +0000 UTC m=+39.027252928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls") pod "image-registry-58d9c8db76-blrq5" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33") : secret "image-registry-tls" not found Jul 19 14:58:06.666726 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.666683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:06.667201 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.666852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:06.667201 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.666895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:06.667201 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.667049 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 14:58:06.667201 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.667113 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls podName:f426dcde-7439-4a0c-8800-f69e4833aa74 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.667094032 +0000 UTC m=+39.531368810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b46dv" (UID: "f426dcde-7439-4a0c-8800-f69e4833aa74") : secret "samples-operator-tls" not found Jul 19 14:58:06.667520 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.667501 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:06.667605 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.667559 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls podName:08f073a5-104f-4805-af4f-afd33d9c109b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.667544417 +0000 UTC m=+39.531819194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-skx69" (UID: "08f073a5-104f-4805-af4f-afd33d9c109b") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:06.667675 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.667641 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:58:06.667728 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.667675 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert podName:508b0e64-758e-4cc0-aa21-eb791dcc95f8 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.66766441 +0000 UTC m=+39.531939191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jwjbv" (UID: "508b0e64-758e-4cc0-aa21-eb791dcc95f8") : secret "networking-console-plugin-cert" not found Jul 19 14:58:06.767444 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.767403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:06.767642 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:06.767457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:06.768476 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.767749 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.767728767 +0000 UTC m=+39.632003558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : configmap references non-existent config key: service-ca.crt Jul 19 14:58:06.768476 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.768218 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:58:06.768476 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:06.768268 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:08.768251991 +0000 UTC m=+39.632526773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : secret "router-metrics-certs-default" not found Jul 19 14:58:07.915645 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:07.915608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:08.182745 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.182649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:08.182917 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.182759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:08.182917 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.182835 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:08.182917 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.182859 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d9c8db76-blrq5: secret "image-registry-tls" not found Jul 19 14:58:08.182917 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.182907 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:08.183159 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.182925 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls podName:6c3bf732-ba93-48f8-afb6-01c2a60f5c33 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.182903238 +0000 UTC m=+43.047178033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls") pod "image-registry-58d9c8db76-blrq5" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33") : secret "image-registry-tls" not found Jul 19 14:58:08.183159 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.182965 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls podName:b6c64568-904b-478f-a9d8-a8b9151a0748 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.182947627 +0000 UTC m=+43.047222411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls") pod "dns-default-ttnr8" (UID: "b6c64568-904b-478f-a9d8-a8b9151a0748") : secret "dns-default-metrics-tls" not found Jul 19 14:58:08.687514 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.687474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:08.687737 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.687591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:08.687737 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.687684 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 14:58:08.687737 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.687709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:08.687892 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.687770 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls podName:f426dcde-7439-4a0c-8800-f69e4833aa74 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.687749282 +0000 UTC m=+43.552024060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b46dv" (UID: "f426dcde-7439-4a0c-8800-f69e4833aa74") : secret "samples-operator-tls" not found Jul 19 14:58:08.687892 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.687818 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:58:08.687892 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.687863 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert podName:508b0e64-758e-4cc0-aa21-eb791dcc95f8 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.687847473 +0000 UTC m=+43.552122248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jwjbv" (UID: "508b0e64-758e-4cc0-aa21-eb791dcc95f8") : secret "networking-console-plugin-cert" not found Jul 19 14:58:08.688033 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.687912 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:08.688033 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.687944 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls podName:08f073a5-104f-4805-af4f-afd33d9c109b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.687933728 +0000 UTC m=+43.552208503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-skx69" (UID: "08f073a5-104f-4805-af4f-afd33d9c109b") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:08.788759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.788714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:08.788759 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.788755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:08.788981 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.788922 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.788898703 +0000 UTC m=+43.653173493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : configmap references non-existent config key: service-ca.crt Jul 19 14:58:08.788981 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.788970 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:58:08.789071 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:08.789024 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.78900882 +0000 UTC m=+43.653283595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : secret "router-metrics-certs-default" not found Jul 19 14:58:08.995180 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:08.995077 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:09.095883 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:09.095844 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:12.219337 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.219298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:12.219804 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.219370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:12.219804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.219447 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:12.219804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.219466 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:12.219804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.219517 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls podName:b6c64568-904b-478f-a9d8-a8b9151a0748 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.219502921 +0000 UTC m=+51.083777697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls") pod "dns-default-ttnr8" (UID: "b6c64568-904b-478f-a9d8-a8b9151a0748") : secret "dns-default-metrics-tls" not found Jul 19 14:58:12.219804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.219466 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d9c8db76-blrq5: secret "image-registry-tls" not found Jul 19 14:58:12.219804 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.219607 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls podName:6c3bf732-ba93-48f8-afb6-01c2a60f5c33 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.219577157 +0000 UTC m=+51.083851935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls") pod "image-registry-58d9c8db76-blrq5" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33") : secret "image-registry-tls" not found Jul 19 14:58:12.723171 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.723131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:12.723359 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.723269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:12.723359 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.723296 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:12.723359 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.723309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:12.723496 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.723364 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls podName:08f073a5-104f-4805-af4f-afd33d9c109b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.723345595 +0000 UTC m=+51.587620373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-skx69" (UID: "08f073a5-104f-4805-af4f-afd33d9c109b") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:12.723496 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.723415 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 14:58:12.723496 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.723435 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:58:12.723496 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.723457 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls podName:f426dcde-7439-4a0c-8800-f69e4833aa74 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.723446323 +0000 UTC m=+51.587721113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b46dv" (UID: "f426dcde-7439-4a0c-8800-f69e4833aa74") : secret "samples-operator-tls" not found Jul 19 14:58:12.723496 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.723497 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert podName:508b0e64-758e-4cc0-aa21-eb791dcc95f8 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.72348078 +0000 UTC m=+51.587755557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jwjbv" (UID: "508b0e64-758e-4cc0-aa21-eb791dcc95f8") : secret "networking-console-plugin-cert" not found Jul 19 14:58:12.824768 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.824731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:12.824768 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:12.824774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:12.825006 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.824886 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:58:12.825006 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.824911 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.824891969 +0000 UTC m=+51.689166754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : configmap references non-existent config key: service-ca.crt Jul 19 14:58:12.825006 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:12.824931 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.82492269 +0000 UTC m=+51.689197465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : secret "router-metrics-certs-default" not found Jul 19 14:58:14.023398 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:14.023356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:14.763713 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:14.763620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:14.976333 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:14.976211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" event={"ID":"b7047527-b902-4be6-94d5-9a3b377122d1","Type":"ContainerStarted","Data":"c244b2c246592fb42c54c11977a9c3ecbcd4ba39783f22d02a7ab7964c14c220"} Jul 19 14:58:14.978307 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:14.978269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" event={"ID":"8fbbbb06-6826-4239-9aea-1909f0723059","Type":"ContainerStarted","Data":"35eaba005475935a8080f005c06ac4193018132ded93e2023ce1d337f9055005"} Jul 19 14:58:15.986655 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.986612 2580 generic.go:356] "Generic (PLEG): container finished" podID="c34f641b-e476-47c0-9dce-506e0d8b2c36" containerID="147170408aa3f605fba4a4cb6bbdec2162a519ddf908b945fae3fb1509205a96" exitCode=0 Jul 19 14:58:15.987081 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.986693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerDied","Data":"147170408aa3f605fba4a4cb6bbdec2162a519ddf908b945fae3fb1509205a96"} Jul 19 14:58:15.989907 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.989366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/0.log" Jul 19 14:58:15.989907 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.989408 2580 generic.go:356] "Generic (PLEG): container finished" podID="f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9" containerID="34d5e7791d31f05508cbc23edbc0a51b8429a06a14d1a5e7276a95e15134052d" exitCode=255 Jul 19 14:58:15.989907 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.989487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-f662r" event={"ID":"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9","Type":"ContainerDied","Data":"34d5e7791d31f05508cbc23edbc0a51b8429a06a14d1a5e7276a95e15134052d"} Jul 19 14:58:15.989907 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.989688 2580 scope.go:117] "RemoveContainer" containerID="34d5e7791d31f05508cbc23edbc0a51b8429a06a14d1a5e7276a95e15134052d" Jul 19 14:58:15.994427 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.994402 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" event={"ID":"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e","Type":"ContainerStarted","Data":"ff36271b3b49493b1f6cc326959f78449b99aa5c1f3a7cfc5e1bf7c10639c087"} Jul 19 14:58:15.995827 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.995805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ndgfw" event={"ID":"83275949-83ec-4fb7-9ca7-7dd6ebe2ce10","Type":"ContainerStarted","Data":"d312a9b8b36a2d8d471efb3c79d51baa433c76cbce8cf20c3c2a2f70060536ca"} Jul 19 14:58:15.997412 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.997394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" event={"ID":"125a2380-9e33-47d6-926a-9af477e93234","Type":"ContainerStarted","Data":"9c37d29772d4909d7e999e1d63d3dc1a84d8cef60cc84406b2ba743be55559f8"} Jul 19 14:58:15.999029 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:15.999012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" event={"ID":"cf3f5226-5ae6-4367-bfbf-6c8f9e2bc238","Type":"ContainerStarted","Data":"4ca8f634f6eacba3e207beb9dcfa6db349620d36a18d0496201ebf1fb63e3a67"} Jul 19 14:58:16.039922 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.039860 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" podStartSLOduration=2.75246362 podStartE2EDuration="12.039836435s" totalImagesPullingTime="9.287372815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:16.030080941 +0000 UTC m=+46.894355738" watchObservedRunningTime="2026-07-19 14:58:16.039836435 +0000 UTC m=+46.904111233" Jul 19 14:58:16.086313 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.086259 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" podStartSLOduration=2.791937218 podStartE2EDuration="12.086239491s" totalImagesPullingTime="9.294302273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:16.083770945 +0000 UTC m=+46.948045759" watchObservedRunningTime="2026-07-19 14:58:16.086239491 +0000 UTC m=+46.950514290" Jul 19 14:58:16.087124 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.086836 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" podStartSLOduration=2.620176679 podStartE2EDuration="12.086826565s" totalImagesPullingTime="9.466649886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:16.06199439 +0000 UTC m=+46.926269183" watchObservedRunningTime="2026-07-19 14:58:16.086826565 +0000 UTC m=+46.951101365" Jul 19 14:58:16.101331 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.101271 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-k7rbl" podStartSLOduration=2.786719325 podStartE2EDuration="12.101248949s" totalImagesPullingTime="9.314529624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:16.099649039 +0000 UTC m=+46.963923862" watchObservedRunningTime="2026-07-19 14:58:16.101248949 +0000 UTC m=+46.965523752" Jul 19 14:58:16.142951 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.142682 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ndgfw" podStartSLOduration=2.238238631 podStartE2EDuration="12.142662714s" totalImagesPullingTime="9.904424083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:16.140889785 +0000 UTC m=+47.005164583" watchObservedRunningTime="2026-07-19 14:58:16.142662714 +0000 UTC m=+47.006937512" Jul 19 14:58:16.160289 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.158801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-m2c5r" podStartSLOduration=2.68128486 podStartE2EDuration="12.158780996s" totalImagesPullingTime="9.477496136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:16.157908251 +0000 UTC m=+47.022183060" watchObservedRunningTime="2026-07-19 14:58:16.158780996 +0000 UTC m=+47.023055793" Jul 19 14:58:16.176977 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.176887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:16.281618 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.281562 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:16.281754 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.281737 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kw4dv"] Jul 19 14:58:16.296599 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.296541 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kw4dv"] Jul 19 14:58:16.296765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.296634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.299513 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.299255 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 19 14:58:16.299513 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.299315 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j74hj\"" Jul 19 14:58:16.299513 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.299415 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 19 14:58:16.340386 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.340352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:16.362429 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.362401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.362601 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.362468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/935ad60d-8783-4dee-bb2c-692896d4e497-data-volume\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.362601 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.362555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.362601 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.362574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt9pl\" (UniqueName: \"kubernetes.io/projected/935ad60d-8783-4dee-bb2c-692896d4e497-kube-api-access-kt9pl\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.362784 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.362646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/935ad60d-8783-4dee-bb2c-692896d4e497-crio-socket\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464019 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.463944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464019 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.464003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/935ad60d-8783-4dee-bb2c-692896d4e497-data-volume\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464198 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.464128 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:16.464240 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.464200 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:16.96418102 +0000 UTC m=+47.828455799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:16.464300 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.464274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464349 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.464303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt9pl\" (UniqueName: \"kubernetes.io/projected/935ad60d-8783-4dee-bb2c-692896d4e497-kube-api-access-kt9pl\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464349 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.464313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/935ad60d-8783-4dee-bb2c-692896d4e497-data-volume\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464435 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.464388 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:16.464483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.464434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/935ad60d-8783-4dee-bb2c-692896d4e497-crio-socket\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464483 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.464385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/935ad60d-8783-4dee-bb2c-692896d4e497-crio-socket\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.464483 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.464453 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:16.964437619 +0000 UTC m=+47.828712416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : configmap "kube-rbac-proxy" not found Jul 19 14:58:16.474882 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.474857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt9pl\" (UniqueName: \"kubernetes.io/projected/935ad60d-8783-4dee-bb2c-692896d4e497-kube-api-access-kt9pl\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.804065 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.803977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:16.880793 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.880762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:16.931874 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.931837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:16.970178 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.970142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.970312 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:16.970260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:16.970312 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.970290 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:16.970407 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.970371 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:16.970407 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.970382 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:17.970367633 +0000 UTC m=+48.834642411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:16.970487 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:16.970418 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:17.970405881 +0000 UTC m=+48.834680659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : configmap "kube-rbac-proxy" not found Jul 19 14:58:17.008151 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.008117 2580 generic.go:356] "Generic (PLEG): container finished" podID="c34f641b-e476-47c0-9dce-506e0d8b2c36" containerID="ea49bab7cebcabf27d4a6b586de0cdfb23838e737d38a65d1aa8dd3351fed20b" exitCode=0 Jul 19 14:58:17.008548 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.008199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerDied","Data":"ea49bab7cebcabf27d4a6b586de0cdfb23838e737d38a65d1aa8dd3351fed20b"} Jul 19 14:58:17.009815 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.009797 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 14:58:17.010234 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.010219 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/0.log" Jul 19 14:58:17.010294 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.010254 2580 generic.go:356] "Generic (PLEG): container finished" podID="f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9" containerID="7362e534cceb87a93cb17ebfc3d196821ee6b0e15d7652a998ff7d9c8f9f5109" exitCode=255 Jul 19 14:58:17.010361 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.010343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-f662r" event={"ID":"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9","Type":"ContainerDied","Data":"7362e534cceb87a93cb17ebfc3d196821ee6b0e15d7652a998ff7d9c8f9f5109"} Jul 19 14:58:17.010407 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.010397 2580 scope.go:117] "RemoveContainer" containerID="34d5e7791d31f05508cbc23edbc0a51b8429a06a14d1a5e7276a95e15134052d" Jul 19 14:58:17.010621 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.010602 2580 scope.go:117] "RemoveContainer" containerID="7362e534cceb87a93cb17ebfc3d196821ee6b0e15d7652a998ff7d9c8f9f5109" Jul 19 14:58:17.010786 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:17.010763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-f662r_openshift-console-operator(f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9)\"" pod="openshift-console-operator/console-operator-565b7b569d-f662r" podUID="f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9" Jul 19 14:58:17.185571 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.185533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:17.286495 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.286464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:17.680277 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.680249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7k7rd_c1e81ea0-1dea-466e-a980-176b3a996a7d/dns-node-resolver/0.log" Jul 19 14:58:17.893568 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.893535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:17.979720 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.979615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:17.979905 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:17.979751 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:17.979905 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.979782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:17.979905 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:17.979811 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:19.979794554 +0000 UTC m=+50.844069329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:17.979905 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:17.979860 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:17.980066 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:17.979912 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:19.97989642 +0000 UTC m=+50.844171196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : configmap "kube-rbac-proxy" not found Jul 19 14:58:17.994245 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:17.994208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:18.017718 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.017682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" event={"ID":"c34f641b-e476-47c0-9dce-506e0d8b2c36","Type":"ContainerStarted","Data":"17b4e79d13c13d8226a95b96c04a4b68ad1383b306f2241ccca9a2c9e479b88c"} Jul 19 14:58:18.019189 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.019173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 14:58:18.019474 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.019459 2580 scope.go:117] "RemoveContainer" containerID="7362e534cceb87a93cb17ebfc3d196821ee6b0e15d7652a998ff7d9c8f9f5109" Jul 19 14:58:18.019622 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:18.019604 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-f662r_openshift-console-operator(f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9)\"" pod="openshift-console-operator/console-operator-565b7b569d-f662r" podUID="f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9" Jul 19 14:58:18.039340 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.039286 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hcjp9" podStartSLOduration=7.8719832499999995 podStartE2EDuration="48.039272166s" totalImagesPullingTime="40.167288916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:18.03880583 +0000 UTC m=+48.903080627" watchObservedRunningTime="2026-07-19 14:58:18.039272166 +0000 UTC m=+48.903546960" Jul 19 14:58:18.133081 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.133045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:18.234526 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.234433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:18.586916 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.586830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:18.680344 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.680316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mscg5_754bd0cf-54de-4f3f-b513-501ec601ec40/node-ca/0.log" Jul 19 14:58:18.687618 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:18.687573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:19.019155 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:19.019120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:19.069895 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:19.069864 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:19.550876 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:19.550845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:19.601875 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:19.601834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:19.680917 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:19.680887 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ndgfw_83275949-83ec-4fb7-9ca7-7dd6ebe2ce10/serve-healthcheck-canary/0.log" Jul 19 14:58:20.000166 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.000135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:20.000321 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.000273 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:20.000321 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.000294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:20.000428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.000353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:24.000336273 +0000 UTC m=+54.864611068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : configmap "kube-rbac-proxy" not found Jul 19 14:58:20.000428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.000422 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:20.000500 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.000471 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:24.000459942 +0000 UTC m=+54.864734728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:20.303047 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.302961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:20.303047 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.303041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:20.303428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.303129 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:20.303428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.303144 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:20.303428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.303148 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d9c8db76-blrq5: secret "image-registry-tls" not found Jul 19 14:58:20.303428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.303195 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls podName:b6c64568-904b-478f-a9d8-a8b9151a0748 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.303181146 +0000 UTC m=+67.167455921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls") pod "dns-default-ttnr8" (UID: "b6c64568-904b-478f-a9d8-a8b9151a0748") : secret "dns-default-metrics-tls" not found Jul 19 14:58:20.303428 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.303207 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls podName:6c3bf732-ba93-48f8-afb6-01c2a60f5c33 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.303201687 +0000 UTC m=+67.167476463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls") pod "image-registry-58d9c8db76-blrq5" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33") : secret "image-registry-tls" not found Jul 19 14:58:20.336155 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.336124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:20.467989 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.467954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:20.807953 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.807922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:20.808136 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.807960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:20.808136 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.808014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:20.808136 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.808079 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 19 14:58:20.808136 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.808119 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:20.808313 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.808138 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 19 14:58:20.808313 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.808146 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert podName:508b0e64-758e-4cc0-aa21-eb791dcc95f8 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.808131233 +0000 UTC m=+67.672406012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jwjbv" (UID: "508b0e64-758e-4cc0-aa21-eb791dcc95f8") : secret "networking-console-plugin-cert" not found Jul 19 14:58:20.808313 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.808197 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls podName:08f073a5-104f-4805-af4f-afd33d9c109b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.808184062 +0000 UTC m=+67.672458838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-skx69" (UID: "08f073a5-104f-4805-af4f-afd33d9c109b") : secret "cluster-monitoring-operator-tls" not found Jul 19 14:58:20.808313 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.808209 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls podName:f426dcde-7439-4a0c-8800-f69e4833aa74 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.80820141 +0000 UTC m=+67.672476185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b46dv" (UID: "f426dcde-7439-4a0c-8800-f69e4833aa74") : secret "samples-operator-tls" not found Jul 19 14:58:20.909419 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.909387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:20.909419 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:20.909422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:20.909635 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.909553 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.909535974 +0000 UTC m=+67.773810748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : configmap references non-existent config key: service-ca.crt Jul 19 14:58:20.909635 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.909561 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 19 14:58:20.909635 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:20.909613 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs podName:17ac597b-4a6e-4a0f-a36e-5914098f469b nodeName:}" failed. No retries permitted until 2026-07-19 14:58:36.909601492 +0000 UTC m=+67.773876267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs") pod "router-default-75784d96f4-xmj78" (UID: "17ac597b-4a6e-4a0f-a36e-5914098f469b") : secret "router-metrics-certs-default" not found Jul 19 14:58:23.286307 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:23.286273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:23.594847 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:23.594761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:23.695998 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:23.695967 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:23.796199 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:23.796160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:23.872960 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:23.872917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:24.009954 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:24.009917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:24.037120 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:24.037096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:24.037264 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:24.037199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:24.037264 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:24.037210 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:24.037353 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:24.037271 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls podName:935ad60d-8783-4dee-bb2c-692896d4e497 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:32.037253201 +0000 UTC m=+62.901527980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kw4dv" (UID: "935ad60d-8783-4dee-bb2c-692896d4e497") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:24.037756 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:24.037728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/935ad60d-8783-4dee-bb2c-692896d4e497-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:24.107682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:24.107631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:24.488717 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:24.488683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:25.090594 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:25.090559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:25.285185 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:25.285154 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:25.285350 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:25.285202 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:25.285537 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:25.285526 2580 scope.go:117] "RemoveContainer" containerID="7362e534cceb87a93cb17ebfc3d196821ee6b0e15d7652a998ff7d9c8f9f5109" Jul 19 14:58:25.285722 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:58:25.285705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-f662r_openshift-console-operator(f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9)\"" pod="openshift-console-operator/console-operator-565b7b569d-f662r" podUID="f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9" Jul 19 14:58:25.487085 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:25.487050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:25.884275 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:25.884235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:26.306627 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:26.306513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:26.487287 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:26.487259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:26.626802 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:26.626767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:26.890821 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:26.890745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:27.206551 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:27.206471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:27.307125 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:27.307086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:27.408015 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:27.407978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:28.088125 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:28.088094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:28.187223 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:28.187186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:28.287139 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:28.287109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:28.682496 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:28.682460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:28.894256 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:28.894215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:29.484998 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:29.484959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:29.784620 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:29.784510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:29.885888 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:29.885846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:30.282047 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:30.282007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:30.485946 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:30.485907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:30.888937 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:30.888904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:30.932822 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:30.932794 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h4h72" Jul 19 14:58:31.486379 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:31.486343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:31.886602 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:31.886548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:32.109979 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.109940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:32.112269 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.112248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/935ad60d-8783-4dee-bb2c-692896d4e497-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw4dv\" (UID: \"935ad60d-8783-4dee-bb2c-692896d4e497\") " pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:32.229871 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.229778 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j74hj\"" Jul 19 14:58:32.238532 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.238515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kw4dv" Jul 19 14:58:32.290032 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.289947 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:32.359209 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.359181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kw4dv"] Jul 19 14:58:32.361947 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:32.361913 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935ad60d_8783_4dee_bb2c_692896d4e497.slice/crio-5f965dadc7b5e698a399dfba218d544b4bccd3a6d06cc84bf6400cd853da5014 WatchSource:0}: Error finding container 5f965dadc7b5e698a399dfba218d544b4bccd3a6d06cc84bf6400cd853da5014: Status 404 returned error can't find the container with id 5f965dadc7b5e698a399dfba218d544b4bccd3a6d06cc84bf6400cd853da5014 Jul 19 14:58:32.484918 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.484825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:32.685258 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:32.685222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:33.069892 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:33.069786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw4dv" event={"ID":"935ad60d-8783-4dee-bb2c-692896d4e497","Type":"ContainerStarted","Data":"5f965dadc7b5e698a399dfba218d544b4bccd3a6d06cc84bf6400cd853da5014"} Jul 19 14:58:33.682716 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:33.682672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:33.882892 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:33.882849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:33.983063 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:33.982985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:34.083718 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.083679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:34.187482 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.187436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:34.288987 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.288890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:34.332849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.332811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:34.332849 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.332854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:34.335283 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.335258 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:58:34.335903 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.335888 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:58:34.345970 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.345945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4142486-9667-4b0c-9304-d6edc4c4d09b-metrics-certs\") pod \"network-metrics-daemon-xpbn5\" (UID: \"a4142486-9667-4b0c-9304-d6edc4c4d09b\") " pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:34.346093 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.345998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63cfb73d-a2d1-41a6-8911-03772f734c2d-original-pull-secret\") pod \"global-pull-secret-syncer-rgb9j\" (UID: \"63cfb73d-a2d1-41a6-8911-03772f734c2d\") " pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:34.370117 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.370091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hc8w7\"" Jul 19 14:58:34.378961 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.378937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpbn5" Jul 19 14:58:34.434300 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.434270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:34.437221 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.437171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25xh\" (UniqueName: \"kubernetes.io/projected/e0b8549c-1b3d-423f-9df2-a5a583a7194c-kube-api-access-q25xh\") pod \"network-check-target-gsvrk\" (UID: \"e0b8549c-1b3d-423f-9df2-a5a583a7194c\") " pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:34.482493 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.482463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgb9j" Jul 19 14:58:34.515968 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.515937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xpbn5"] Jul 19 14:58:34.677226 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.677197 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7fqw\"" Jul 19 14:58:34.685635 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:34.685611 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:36.352840 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.352801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:36.353209 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.352881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:36.355129 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.355106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"image-registry-58d9c8db76-blrq5\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:36.355238 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.355216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6c64568-904b-478f-a9d8-a8b9151a0748-metrics-tls\") pod \"dns-default-ttnr8\" (UID: \"b6c64568-904b-478f-a9d8-a8b9151a0748\") " pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:36.445980 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:36.445943 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4142486_9667_4b0c_9304_d6edc4c4d09b.slice/crio-5ff23256aa650f55c0d2785ee465f768a5a04899c04f687a697e28ead2b16953 WatchSource:0}: Error finding container 5ff23256aa650f55c0d2785ee465f768a5a04899c04f687a697e28ead2b16953: Status 404 returned error can't find the container with id 5ff23256aa650f55c0d2785ee465f768a5a04899c04f687a697e28ead2b16953 Jul 19 14:58:36.531164 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.531130 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-znqlg\"" Jul 19 14:58:36.539840 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.539812 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:36.543901 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.543872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvk2r\"" Jul 19 14:58:36.552615 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.552594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:36.856610 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.856565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:36.856781 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.856693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:36.856781 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.856735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:36.859379 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.859350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f426dcde-7439-4a0c-8800-f69e4833aa74-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b46dv\" (UID: \"f426dcde-7439-4a0c-8800-f69e4833aa74\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:36.859515 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.859436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/08f073a5-104f-4805-af4f-afd33d9c109b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-skx69\" (UID: \"08f073a5-104f-4805-af4f-afd33d9c109b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:36.859829 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.859809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/508b0e64-758e-4cc0-aa21-eb791dcc95f8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jwjbv\" (UID: \"508b0e64-758e-4cc0-aa21-eb791dcc95f8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:36.958045 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.957999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:36.958236 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.958054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:36.958742 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.958716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17ac597b-4a6e-4a0f-a36e-5914098f469b-service-ca-bundle\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:36.960635 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.960611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ac597b-4a6e-4a0f-a36e-5914098f469b-metrics-certs\") pod \"router-default-75784d96f4-xmj78\" (UID: \"17ac597b-4a6e-4a0f-a36e-5914098f469b\") " pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:36.999999 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:36.999971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z2gd4\"" Jul 19 14:58:37.007735 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.007709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" Jul 19 14:58:37.007884 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.007763 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tf9jj\"" Jul 19 14:58:37.013680 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.013656 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q54d7\"" Jul 19 14:58:37.016184 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.016169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" Jul 19 14:58:37.022954 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.022936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" Jul 19 14:58:37.088495 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.088453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpbn5" event={"ID":"a4142486-9667-4b0c-9304-d6edc4c4d09b","Type":"ContainerStarted","Data":"5ff23256aa650f55c0d2785ee465f768a5a04899c04f687a697e28ead2b16953"} Jul 19 14:58:37.096177 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.096147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tvzzw\"" Jul 19 14:58:37.104208 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.104173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:37.372747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.371243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rgb9j"] Jul 19 14:58:37.439466 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.439430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv"] Jul 19 14:58:37.441304 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:37.441277 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508b0e64_758e_4cc0_aa21_eb791dcc95f8.slice/crio-0144338cd39fc26a637d2d9c375c252abed264ea9a52548de26dfc60bc242f08 WatchSource:0}: Error finding container 0144338cd39fc26a637d2d9c375c252abed264ea9a52548de26dfc60bc242f08: Status 404 returned error can't find the container with id 0144338cd39fc26a637d2d9c375c252abed264ea9a52548de26dfc60bc242f08 Jul 19 14:58:37.443174 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.443152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gsvrk"] Jul 19 14:58:37.447290 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.446985 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d9c8db76-blrq5"] Jul 19 14:58:37.454419 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:37.454239 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b8549c_1b3d_423f_9df2_a5a583a7194c.slice/crio-342d6ccf63e2aaaf207b6d9f29e17ae9369246b88616f9c13e219051c9873658 WatchSource:0}: Error finding container 342d6ccf63e2aaaf207b6d9f29e17ae9369246b88616f9c13e219051c9873658: Status 404 returned error can't find the container with id 342d6ccf63e2aaaf207b6d9f29e17ae9369246b88616f9c13e219051c9873658 Jul 19 14:58:37.454721 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:37.454682 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3bf732_ba93_48f8_afb6_01c2a60f5c33.slice/crio-a5724c4374014f58f1dede7c8140da94b386b8980071244b1538e4e54c69c12b WatchSource:0}: Error finding container a5724c4374014f58f1dede7c8140da94b386b8980071244b1538e4e54c69c12b: Status 404 returned error can't find the container with id a5724c4374014f58f1dede7c8140da94b386b8980071244b1538e4e54c69c12b Jul 19 14:58:37.470079 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:37.470019 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f073a5_104f_4805_af4f_afd33d9c109b.slice/crio-2dd88e1d895b9fd65ed9febb529f320a393130e3de8d0817c7016571fc56d575 WatchSource:0}: Error finding container 2dd88e1d895b9fd65ed9febb529f320a393130e3de8d0817c7016571fc56d575: Status 404 returned error can't find the container with id 2dd88e1d895b9fd65ed9febb529f320a393130e3de8d0817c7016571fc56d575 Jul 19 14:58:37.470079 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.470073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69"] Jul 19 14:58:37.470737 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:37.470705 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c64568_904b_478f_a9d8_a8b9151a0748.slice/crio-6fc6d831efd625c1157d6fe22c41dc48ff698aaf9933079ed1213c7de08fa5b0 WatchSource:0}: Error finding container 6fc6d831efd625c1157d6fe22c41dc48ff698aaf9933079ed1213c7de08fa5b0: Status 404 returned error can't find the container with id 6fc6d831efd625c1157d6fe22c41dc48ff698aaf9933079ed1213c7de08fa5b0 Jul 19 14:58:37.471453 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.471386 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ttnr8"] Jul 19 14:58:37.656807 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.656732 2580 scope.go:117] "RemoveContainer" containerID="7362e534cceb87a93cb17ebfc3d196821ee6b0e15d7652a998ff7d9c8f9f5109" Jul 19 14:58:37.693102 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.693073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv"] Jul 19 14:58:37.696132 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:37.696009 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75784d96f4-xmj78"] Jul 19 14:58:37.698779 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:37.698749 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ac597b_4a6e_4a0f_a36e_5914098f469b.slice/crio-7ef78785c126053a2655740c248bfc49699df9d6a88abcf6735f5e7d7cc61ae4 WatchSource:0}: Error finding container 7ef78785c126053a2655740c248bfc49699df9d6a88abcf6735f5e7d7cc61ae4: Status 404 returned error can't find the container with id 7ef78785c126053a2655740c248bfc49699df9d6a88abcf6735f5e7d7cc61ae4 Jul 19 14:58:38.100444 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.100418 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 14:58:38.100544 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.100516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-f662r" event={"ID":"f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9","Type":"ContainerStarted","Data":"c935ee0109c7863d2f4bdff4554aee6bee8de53cac3239986553ccf4db1a4117"} Jul 19 14:58:38.102725 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.102702 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:38.105985 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.105916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpbn5" event={"ID":"a4142486-9667-4b0c-9304-d6edc4c4d09b","Type":"ContainerStarted","Data":"4d604fbd5ce264cca50f5f2b42e5570a27f17a580f6b129ab98f60ad171ce6b9"} Jul 19 14:58:38.109558 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.109000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75784d96f4-xmj78" event={"ID":"17ac597b-4a6e-4a0f-a36e-5914098f469b","Type":"ContainerStarted","Data":"64da2cb9e863b233d04293729929a8f3bc8bda698334425cad5100e665c5dd8f"} Jul 19 14:58:38.109558 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.109053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75784d96f4-xmj78" event={"ID":"17ac597b-4a6e-4a0f-a36e-5914098f469b","Type":"ContainerStarted","Data":"7ef78785c126053a2655740c248bfc49699df9d6a88abcf6735f5e7d7cc61ae4"} Jul 19 14:58:38.113230 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.113147 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" event={"ID":"f426dcde-7439-4a0c-8800-f69e4833aa74","Type":"ContainerStarted","Data":"c76a6a06becd50f21d6a3b5a483d0681f536bb6c36e8c7f5804975c19c33c23b"} Jul 19 14:58:38.118536 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.118453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ttnr8" event={"ID":"b6c64568-904b-478f-a9d8-a8b9151a0748","Type":"ContainerStarted","Data":"6fc6d831efd625c1157d6fe22c41dc48ff698aaf9933079ed1213c7de08fa5b0"} Jul 19 14:58:38.124819 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.124125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" event={"ID":"6c3bf732-ba93-48f8-afb6-01c2a60f5c33","Type":"ContainerStarted","Data":"6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954"} Jul 19 14:58:38.124819 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.124173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" event={"ID":"6c3bf732-ba93-48f8-afb6-01c2a60f5c33","Type":"ContainerStarted","Data":"a5724c4374014f58f1dede7c8140da94b386b8980071244b1538e4e54c69c12b"} Jul 19 14:58:38.124819 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.124778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:58:38.137032 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.135496 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-f662r" podStartSLOduration=24.804550707 podStartE2EDuration="34.135475711s" totalImagesPullingTime="9.330925004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:38.117444021 +0000 UTC m=+68.981718818" watchObservedRunningTime="2026-07-19 14:58:38.135475711 +0000 UTC m=+68.999750509" Jul 19 14:58:38.137032 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.135854 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75784d96f4-xmj78" podStartSLOduration=34.135844043 podStartE2EDuration="34.135844043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:38.134424428 +0000 UTC m=+68.998699256" watchObservedRunningTime="2026-07-19 14:58:38.135844043 +0000 UTC m=+69.000118841" Jul 19 14:58:38.137032 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.136304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw4dv" event={"ID":"935ad60d-8783-4dee-bb2c-692896d4e497","Type":"ContainerStarted","Data":"80111bf6b3a2b736e3fb868569cb78a13ee55fea2d7c2ee8652c7dc55ec50b98"} Jul 19 14:58:38.142747 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.142683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" event={"ID":"08f073a5-104f-4805-af4f-afd33d9c109b","Type":"ContainerStarted","Data":"2dd88e1d895b9fd65ed9febb529f320a393130e3de8d0817c7016571fc56d575"} Jul 19 14:58:38.149379 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.149338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" event={"ID":"508b0e64-758e-4cc0-aa21-eb791dcc95f8","Type":"ContainerStarted","Data":"0144338cd39fc26a637d2d9c375c252abed264ea9a52548de26dfc60bc242f08"} Jul 19 14:58:38.153419 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.152489 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" podStartSLOduration=68.152472179 podStartE2EDuration="1m8.152472179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:38.1522837 +0000 UTC m=+69.016558499" watchObservedRunningTime="2026-07-19 14:58:38.152472179 +0000 UTC m=+69.016746973" Jul 19 14:58:38.164601 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.164459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gsvrk" event={"ID":"e0b8549c-1b3d-423f-9df2-a5a583a7194c","Type":"ContainerStarted","Data":"5743fcce9b032223d6b369419f68d822822d0fea763a01578ef3952cef7f8506"} Jul 19 14:58:38.164601 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.164543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gsvrk" event={"ID":"e0b8549c-1b3d-423f-9df2-a5a583a7194c","Type":"ContainerStarted","Data":"342d6ccf63e2aaaf207b6d9f29e17ae9369246b88616f9c13e219051c9873658"} Jul 19 14:58:38.165598 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.165523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:58:38.174143 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:38.174111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rgb9j" event={"ID":"63cfb73d-a2d1-41a6-8911-03772f734c2d","Type":"ContainerStarted","Data":"aecbce1d9b52d1c2acb1640430e8048342e4b2f80b16d8312799ee399ea4112e"} Jul 19 14:58:39.101927 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.101471 2580 patch_prober.go:28] interesting pod/console-operator-565b7b569d-f662r container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 19 14:58:39.101927 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.101547 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-f662r" podUID="f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 19 14:58:39.104965 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.104938 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:39.108100 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.108078 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:39.128808 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.128747 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gsvrk" podStartSLOduration=70.128723544 podStartE2EDuration="1m10.128723544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:38.184682549 +0000 UTC m=+69.048957347" watchObservedRunningTime="2026-07-19 14:58:39.128723544 +0000 UTC m=+69.992998322" Jul 19 14:58:39.188233 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.188194 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpbn5" event={"ID":"a4142486-9667-4b0c-9304-d6edc4c4d09b","Type":"ContainerStarted","Data":"145a58f38c3d993ce0b0c8a64b0326b6c4c9c5e78935bd12e6696bc1a7c46b66"} Jul 19 14:58:39.188411 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.188262 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:39.188970 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.188950 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75784d96f4-xmj78" Jul 19 14:58:39.208439 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.208256 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xpbn5" podStartSLOduration=67.669044091 podStartE2EDuration="1m9.208233497s" totalImagesPullingTime="1.539189406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:39.206314669 +0000 UTC m=+70.070589485" watchObservedRunningTime="2026-07-19 14:58:39.208233497 +0000 UTC m=+70.072508294" Jul 19 14:58:39.483534 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.483301 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-f662r" Jul 19 14:58:39.670162 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.670080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:39.778278 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.778191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:39.953140 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:39.953078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:40.137485 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:40.137447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:40.528622 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:40.528539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:40.580295 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:40.580250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:40.744258 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:40.744162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:40.964038 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:40.964002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:41.148619 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.148560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:41.356393 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.356354 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s"] Jul 19 14:58:41.377164 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.375766 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s"] Jul 19 14:58:41.377164 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.375884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.378773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.378680 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 19 14:58:41.378773 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.378689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:41.380563 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.379400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 19 14:58:41.380563 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.379439 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 19 14:58:41.380563 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.379791 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 19 14:58:41.380563 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.380116 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nkk2l\"" Jul 19 14:58:41.439046 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.439010 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-6chsd"] Jul 19 14:58:41.444248 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.444224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:58:41.446628 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.446380 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 19 14:58:41.446628 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.446494 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 19 14:58:41.447275 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.447079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xspdl\"" Jul 19 14:58:41.453768 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.453736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-6chsd"] Jul 19 14:58:41.475609 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.475540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:41.508448 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.508415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/730693f7-27bc-4ce2-b1d6-115a26f0e294-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c7d575bcd-rq78s\" (UID: \"730693f7-27bc-4ce2-b1d6-115a26f0e294\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.508644 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.508473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjpm\" (UniqueName: \"kubernetes.io/projected/730693f7-27bc-4ce2-b1d6-115a26f0e294-kube-api-access-gqjpm\") pod \"managed-serviceaccount-addon-agent-5c7d575bcd-rq78s\" (UID: \"730693f7-27bc-4ce2-b1d6-115a26f0e294\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.609982 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.609917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/730693f7-27bc-4ce2-b1d6-115a26f0e294-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c7d575bcd-rq78s\" (UID: \"730693f7-27bc-4ce2-b1d6-115a26f0e294\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.610233 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.610004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjpm\" (UniqueName: \"kubernetes.io/projected/730693f7-27bc-4ce2-b1d6-115a26f0e294-kube-api-access-gqjpm\") pod \"managed-serviceaccount-addon-agent-5c7d575bcd-rq78s\" (UID: \"730693f7-27bc-4ce2-b1d6-115a26f0e294\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.610233 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.610053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clpj\" (UniqueName: \"kubernetes.io/projected/71ca793f-d8d5-49bf-a27c-6c821e39fcea-kube-api-access-6clpj\") pod \"downloads-5bc67dcd46-6chsd\" (UID: \"71ca793f-d8d5-49bf-a27c-6c821e39fcea\") " pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:58:41.613230 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.613204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/730693f7-27bc-4ce2-b1d6-115a26f0e294-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c7d575bcd-rq78s\" (UID: \"730693f7-27bc-4ce2-b1d6-115a26f0e294\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.622239 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.622211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjpm\" (UniqueName: \"kubernetes.io/projected/730693f7-27bc-4ce2-b1d6-115a26f0e294-kube-api-access-gqjpm\") pod \"managed-serviceaccount-addon-agent-5c7d575bcd-rq78s\" (UID: \"730693f7-27bc-4ce2-b1d6-115a26f0e294\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.648557 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.648510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:41.698414 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.698329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" Jul 19 14:58:41.711313 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.711289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clpj\" (UniqueName: \"kubernetes.io/projected/71ca793f-d8d5-49bf-a27c-6c821e39fcea-kube-api-access-6clpj\") pod \"downloads-5bc67dcd46-6chsd\" (UID: \"71ca793f-d8d5-49bf-a27c-6c821e39fcea\") " pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:58:41.719059 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.719036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clpj\" (UniqueName: \"kubernetes.io/projected/71ca793f-d8d5-49bf-a27c-6c821e39fcea-kube-api-access-6clpj\") pod \"downloads-5bc67dcd46-6chsd\" (UID: \"71ca793f-d8d5-49bf-a27c-6c821e39fcea\") " pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:58:41.747376 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.747333 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:41.757119 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.757093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:58:41.817100 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:41.817063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:43.363221 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:43.363184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:43.416941 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:43.416907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:46.100195 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.100120 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-6chsd"] Jul 19 14:58:46.102489 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:46.102456 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ca793f_d8d5_49bf_a27c_6c821e39fcea.slice/crio-c20baecc274df7de34d82f9e69fccdc935cbe9a837c37bddaaf574d61c274d1d WatchSource:0}: Error finding container c20baecc274df7de34d82f9e69fccdc935cbe9a837c37bddaaf574d61c274d1d: Status 404 returned error can't find the container with id c20baecc274df7de34d82f9e69fccdc935cbe9a837c37bddaaf574d61c274d1d Jul 19 14:58:46.119352 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.119299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s"] Jul 19 14:58:46.124798 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:58:46.124761 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod730693f7_27bc_4ce2_b1d6_115a26f0e294.slice/crio-945cbfe443842dce2506c7d32e46c003adfa15eeee120884ec89f1df81feb8a7 WatchSource:0}: Error finding container 945cbfe443842dce2506c7d32e46c003adfa15eeee120884ec89f1df81feb8a7: Status 404 returned error can't find the container with id 945cbfe443842dce2506c7d32e46c003adfa15eeee120884ec89f1df81feb8a7 Jul 19 14:58:46.222865 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.222825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" event={"ID":"08f073a5-104f-4805-af4f-afd33d9c109b","Type":"ContainerStarted","Data":"c050d25189947fb1e296a8b58d6b0d4ce6b05ea9dcc1002e83c5df2df1bf7a63"} Jul 19 14:58:46.228156 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.227652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" event={"ID":"508b0e64-758e-4cc0-aa21-eb791dcc95f8","Type":"ContainerStarted","Data":"9c587811628f0d1c1fd2f4c28f8b0ddedb04948c3ad86575a352b17269cffebc"} Jul 19 14:58:46.230210 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.230174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" event={"ID":"f426dcde-7439-4a0c-8800-f69e4833aa74","Type":"ContainerStarted","Data":"bc81f298ebb24c0741f21375164a97ee875eef2a76d9390692cdba762681f7db"} Jul 19 14:58:46.232384 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.232333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ttnr8" event={"ID":"b6c64568-904b-478f-a9d8-a8b9151a0748","Type":"ContainerStarted","Data":"2b1d75559013f6a6bdce1ae015f85a4db677945e9f82abdcccb62eb6ed8df650"} Jul 19 14:58:46.234060 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.234018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-6chsd" event={"ID":"71ca793f-d8d5-49bf-a27c-6c821e39fcea","Type":"ContainerStarted","Data":"c20baecc274df7de34d82f9e69fccdc935cbe9a837c37bddaaf574d61c274d1d"} Jul 19 14:58:46.235957 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.235917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" event={"ID":"730693f7-27bc-4ce2-b1d6-115a26f0e294","Type":"ContainerStarted","Data":"945cbfe443842dce2506c7d32e46c003adfa15eeee120884ec89f1df81feb8a7"} Jul 19 14:58:46.239949 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.239556 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-skx69" podStartSLOduration=33.756412275 podStartE2EDuration="42.239537734s" totalImagesPullingTime="8.483125459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:46.23811389 +0000 UTC m=+77.102388689" watchObservedRunningTime="2026-07-19 14:58:46.239537734 +0000 UTC m=+77.103812532" Jul 19 14:58:46.251890 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.251843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jwjbv" podStartSLOduration=33.93838495 podStartE2EDuration="42.251826963s" totalImagesPullingTime="8.313442013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:46.2515642 +0000 UTC m=+77.115839000" watchObservedRunningTime="2026-07-19 14:58:46.251826963 +0000 UTC m=+77.116101741" Jul 19 14:58:46.385159 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.385091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:46.458963 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.458390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:46.576915 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.576817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:46.640144 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.640108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:46.696100 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.696064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:46.790555 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:46.790517 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:47.245435 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.245374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" event={"ID":"f426dcde-7439-4a0c-8800-f69e4833aa74","Type":"ContainerStarted","Data":"1ca4408ce24869846279cab6235de30820afbbf687c51184f3e5c895b8fc5e97"} Jul 19 14:58:47.249560 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.249342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ttnr8" event={"ID":"b6c64568-904b-478f-a9d8-a8b9151a0748","Type":"ContainerStarted","Data":"ef8d34ca57648b773d3aa64f5666063a949105c60918c308e5133cb763c0e842"} Jul 19 14:58:47.249769 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.249729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:47.252566 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.252537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw4dv" event={"ID":"935ad60d-8783-4dee-bb2c-692896d4e497","Type":"ContainerStarted","Data":"1bb11a50b0b4e8f1189399e097395500fb1a00ede2da33877c20bebd646ef366"} Jul 19 14:58:47.254700 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.254673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rgb9j" event={"ID":"63cfb73d-a2d1-41a6-8911-03772f734c2d","Type":"ContainerStarted","Data":"67fd60a07add9966d3e77329dc13ee63f006459cd80ed80e4100f32473b1cd38"} Jul 19 14:58:47.262621 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.262548 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b46dv" podStartSLOduration=35.031445929 podStartE2EDuration="43.262528038s" totalImagesPullingTime="8.231082109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:47.261600679 +0000 UTC m=+78.125875475" watchObservedRunningTime="2026-07-19 14:58:47.262528038 +0000 UTC m=+78.126802836" Jul 19 14:58:47.296723 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.296662 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ttnr8" podStartSLOduration=34.815240122 podStartE2EDuration="43.296639728s" totalImagesPullingTime="8.481399606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:47.296570475 +0000 UTC m=+78.160845273" watchObservedRunningTime="2026-07-19 14:58:47.296639728 +0000 UTC m=+78.160914525" Jul 19 14:58:47.297370 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:47.297331 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rgb9j" podStartSLOduration=68.723410354 podStartE2EDuration="1m17.297318302s" totalImagesPullingTime="8.573907948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:47.280832465 +0000 UTC m=+78.145107269" watchObservedRunningTime="2026-07-19 14:58:47.297318302 +0000 UTC m=+78.161593104" Jul 19 14:58:48.971543 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:48.971507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:49.027243 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:49.027068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:49.500209 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:49.500160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:49.601261 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:49.601226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:51.170284 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:51.170227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:51.222030 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:51.221991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:52.053430 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.053292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:52.105877 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.105840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:52.285450 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.285412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" event={"ID":"730693f7-27bc-4ce2-b1d6-115a26f0e294","Type":"ContainerStarted","Data":"e21b75667f4a413ad26e92ce3a2b7620e3ce573428087ca73a6b7d43e897d23c"} Jul 19 14:58:52.287801 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.287771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw4dv" event={"ID":"935ad60d-8783-4dee-bb2c-692896d4e497","Type":"ContainerStarted","Data":"ab69a52e650223554083bc3c8ff889de2922025a5a403cef09145494308c03cd"} Jul 19 14:58:52.302741 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.302686 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c7d575bcd-rq78s" podStartSLOduration=5.955659958 podStartE2EDuration="11.302667583s" totalImagesPullingTime="5.347007625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:52.300554069 +0000 UTC m=+83.164828900" watchObservedRunningTime="2026-07-19 14:58:52.302667583 +0000 UTC m=+83.166942371" Jul 19 14:58:52.319900 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.319806 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kw4dv" podStartSLOduration=17.504537636 podStartE2EDuration="36.319786234s" totalImagesPullingTime="18.815248598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:58:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:52.318173869 +0000 UTC m=+83.182448671" watchObservedRunningTime="2026-07-19 14:58:52.319786234 +0000 UTC m=+83.184061036" Jul 19 14:58:52.941456 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:52.941415 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:53.034769 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:53.034735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:54.609765 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:54.609734 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:54.685894 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:54.685858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:54.962096 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:54.961988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:55.744555 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:55.744512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:55.845825 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:55.845785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:57.262776 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:57.262741 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ttnr8" Jul 19 14:58:58.577140 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:58.577099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:58.678399 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:58.678365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:59.001642 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:59.001574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:59.056392 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:59.056351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:59.116198 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:59.116162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:59.437967 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:59.437916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:59.604132 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:59.604052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:58:59.655469 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:58:59.655433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:00.198093 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:00.198065 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:59:00.393802 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:00.393754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:00.483520 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:00.483420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:01.752423 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:01.752389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:01.954010 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:01.953965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:01.998305 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:01.998269 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bdk76"] Jul 19 14:59:02.054508 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.054391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.111208 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.111156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.180096 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.180063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.183375 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.183339 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 19 14:59:02.183977 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.183953 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 19 14:59:02.184206 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.184188 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 19 14:59:02.184696 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.184678 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 19 14:59:02.185447 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.185427 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tcft9\"" Jul 19 14:59:02.249144 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.249108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.288411 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khx4d\" (UniqueName: \"kubernetes.io/projected/6b3b9ec8-6733-48c2-8ec8-97b468132c01-kube-api-access-khx4d\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288411 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-root\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288652 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-tls\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288652 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b9ec8-6733-48c2-8ec8-97b468132c01-metrics-client-ca\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288652 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-sys\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288793 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288793 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288793 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-wtmp\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.288793 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.288791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-textfile\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390299 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-textfile\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khx4d\" (UniqueName: \"kubernetes.io/projected/6b3b9ec8-6733-48c2-8ec8-97b468132c01-kube-api-access-khx4d\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-root\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-tls\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b9ec8-6733-48c2-8ec8-97b468132c01-metrics-client-ca\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-sys\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-wtmp\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-sys\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-root\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-textfile\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.390833 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.390786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-wtmp\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.391325 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.391261 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.391472 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.391333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b9ec8-6733-48c2-8ec8-97b468132c01-metrics-client-ca\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.393492 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.393466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.393630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.393552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b3b9ec8-6733-48c2-8ec8-97b468132c01-node-exporter-tls\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.398451 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.398426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khx4d\" (UniqueName: \"kubernetes.io/projected/6b3b9ec8-6733-48c2-8ec8-97b468132c01-kube-api-access-khx4d\") pod \"node-exporter-bdk76\" (UID: \"6b3b9ec8-6733-48c2-8ec8-97b468132c01\") " pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.402417 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.402392 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.496159 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.496125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdk76" Jul 19 14:59:02.500204 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.500176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.602904 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.602861 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.703819 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.703708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:02.966877 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:02.966736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:03.037797 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:03.037761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:03.148214 ip-10-0-140-163 kubenswrapper[2580]: W0719 14:59:03.148181 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3b9ec8_6733_48c2_8ec8_97b468132c01.slice/crio-2676de8a979858252ed53606398bfaca4254a1c9b4215f605d983d4f0727406b WatchSource:0}: Error finding container 2676de8a979858252ed53606398bfaca4254a1c9b4215f605d983d4f0727406b: Status 404 returned error can't find the container with id 2676de8a979858252ed53606398bfaca4254a1c9b4215f605d983d4f0727406b Jul 19 14:59:03.233440 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:03.233355 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:03.337425 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:03.337393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdk76" event={"ID":"6b3b9ec8-6733-48c2-8ec8-97b468132c01","Type":"ContainerStarted","Data":"2676de8a979858252ed53606398bfaca4254a1c9b4215f605d983d4f0727406b"} Jul 19 14:59:04.344033 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:04.343995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-6chsd" event={"ID":"71ca793f-d8d5-49bf-a27c-6c821e39fcea","Type":"ContainerStarted","Data":"932c16762f147f1cada6032fb877d2e31168a8d63f37571b5efc9de68c215ba4"} Jul 19 14:59:04.344528 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:04.344250 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:59:04.355825 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:04.355796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-6chsd" Jul 19 14:59:04.360897 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:04.360839 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-6chsd" podStartSLOduration=6.063994969 podStartE2EDuration="23.360821308s" totalImagesPullingTime="17.296826339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:04.358822249 +0000 UTC m=+95.223097052" watchObservedRunningTime="2026-07-19 14:59:04.360821308 +0000 UTC m=+95.225096104" Jul 19 14:59:04.916197 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:04.916157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:04.986250 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:04.986215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:05.046106 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.046069 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d9c8db76-blrq5"] Jul 19 14:59:05.128242 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.128207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:05.180191 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.180107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:05.282295 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.282255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:05.350520 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.350484 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b3b9ec8-6733-48c2-8ec8-97b468132c01" containerID="65477e0266e824f917d5c2e404aaf8a35d58e57112c1eadecfa8b3165b5ca95a" exitCode=0 Jul 19 14:59:05.351022 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.350568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdk76" event={"ID":"6b3b9ec8-6733-48c2-8ec8-97b468132c01","Type":"ContainerDied","Data":"65477e0266e824f917d5c2e404aaf8a35d58e57112c1eadecfa8b3165b5ca95a"} Jul 19 14:59:05.438417 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:05.438320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:06.358350 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:06.358303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdk76" event={"ID":"6b3b9ec8-6733-48c2-8ec8-97b468132c01","Type":"ContainerStarted","Data":"c374b13b5057ebc19491615cd2c66ff7f8fd027d62e0344a088ab7f4f23b67cf"} Jul 19 14:59:06.358822 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:06.358367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdk76" event={"ID":"6b3b9ec8-6733-48c2-8ec8-97b468132c01","Type":"ContainerStarted","Data":"668a8e7731bc6ca3b76d03c8c3572872efc8db33382c665d1508d578db1dbd0b"} Jul 19 14:59:06.377617 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:06.377548 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bdk76" podStartSLOduration=3.865512628 podStartE2EDuration="5.377531863s" totalImagesPullingTime="1.393571854s" totalInitContainerRuntime="118.447381ms" isStatefulPod=true podCreationTimestamp="2026-07-19 14:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:06.376128569 +0000 UTC m=+97.240403365" watchObservedRunningTime="2026-07-19 14:59:06.377531863 +0000 UTC m=+97.241806714" Jul 19 14:59:06.390645 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:06.390611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:06.770891 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:06.770796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:06.824422 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:06.824381 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:07.019059 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:07.019019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:07.245945 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:07.245909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:07.348258 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:07.348220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:07.768930 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:07.768877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:07.822257 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:07.822219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:08.216248 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:08.216208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:09.105549 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:09.105506 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:09.175964 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:09.175925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:09.385218 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:09.385185 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:09.477652 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:09.477616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:10.194771 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:10.194738 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gsvrk" Jul 19 14:59:10.374178 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:10.374137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:10.424625 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:10.424566 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:15.386086 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:15.386045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:18.966714 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:18.966675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:19.017007 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:19.016972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:20.375241 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:20.375202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:20.475945 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:20.475905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:21.420627 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:21.420562 2580 generic.go:356] "Generic (PLEG): container finished" podID="bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e" containerID="ff36271b3b49493b1f6cc326959f78449b99aa5c1f3a7cfc5e1bf7c10639c087" exitCode=0 Jul 19 14:59:21.421025 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:21.420645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" event={"ID":"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e","Type":"ContainerDied","Data":"ff36271b3b49493b1f6cc326959f78449b99aa5c1f3a7cfc5e1bf7c10639c087"} Jul 19 14:59:21.421025 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:21.420988 2580 scope.go:117] "RemoveContainer" containerID="ff36271b3b49493b1f6cc326959f78449b99aa5c1f3a7cfc5e1bf7c10639c087" Jul 19 14:59:22.430283 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:22.429488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-sb2j7" event={"ID":"bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e","Type":"ContainerStarted","Data":"f118f4354e0812402da1b78abb98e901c1a091aaed9d87d8ea51c498957a293d"} Jul 19 14:59:22.628373 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:22.628345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75784d96f4-xmj78_17ac597b-4a6e-4a0f-a36e-5914098f469b/router/0.log" Jul 19 14:59:22.644459 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:22.644432 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ndgfw_83275949-83ec-4fb7-9ca7-7dd6ebe2ce10/serve-healthcheck-canary/0.log" Jul 19 14:59:30.074034 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.073985 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" podUID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" containerName="registry" containerID="cri-o://6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954" gracePeriod=30 Jul 19 14:59:30.193462 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.193430 2580 patch_prober.go:28] interesting pod/image-registry-58d9c8db76-blrq5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" start-of-body= Jul 19 14:59:30.193630 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.193488 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" podUID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" Jul 19 14:59:30.360084 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.360055 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:59:30.458084 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.458047 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" containerID="6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954" exitCode=0 Jul 19 14:59:30.458243 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.458091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" event={"ID":"6c3bf732-ba93-48f8-afb6-01c2a60f5c33","Type":"ContainerDied","Data":"6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954"} Jul 19 14:59:30.458243 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.458104 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" Jul 19 14:59:30.458243 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.458159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d9c8db76-blrq5" event={"ID":"6c3bf732-ba93-48f8-afb6-01c2a60f5c33","Type":"ContainerDied","Data":"a5724c4374014f58f1dede7c8140da94b386b8980071244b1538e4e54c69c12b"} Jul 19 14:59:30.458243 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.458180 2580 scope.go:117] "RemoveContainer" containerID="6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954" Jul 19 14:59:30.466032 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.466004 2580 scope.go:117] "RemoveContainer" containerID="6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954" Jul 19 14:59:30.466326 ip-10-0-140-163 kubenswrapper[2580]: E0719 14:59:30.466303 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954\": container with ID starting with 6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954 not found: ID does not exist" containerID="6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954" Jul 19 14:59:30.466394 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.466334 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954"} err="failed to get container status \"6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954\": rpc error: code = NotFound desc = could not find container \"6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954\": container with ID starting with 6e818f56d534723608fb8f5fed2b15eb7798e8298ca82005c7b1fc589ec7f954 not found: ID does not exist" Jul 19 14:59:30.528943 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.528907 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-installation-pull-secrets\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529123 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.528955 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmgq7\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-kube-api-access-rmgq7\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529123 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529004 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-certificates\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529236 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529126 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529236 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529199 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-bound-sa-token\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529353 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529261 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-ca-trust-extracted\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529353 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529288 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-trusted-ca\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529353 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529333 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-image-registry-private-configuration\") pod \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\" (UID: \"6c3bf732-ba93-48f8-afb6-01c2a60f5c33\") " Jul 19 14:59:30.529682 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529655 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:30.529811 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.529755 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:30.531650 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.531559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-kube-api-access-rmgq7" (OuterVolumeSpecName: "kube-api-access-rmgq7") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "kube-api-access-rmgq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:30.531650 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.531567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:30.531887 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.531863 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:30.531940 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.531840 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:30.532011 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.531987 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:30.539973 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.539942 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6c3bf732-ba93-48f8-afb6-01c2a60f5c33" (UID: "6c3bf732-ba93-48f8-afb6-01c2a60f5c33"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:59:30.630781 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630742 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-image-registry-private-configuration\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.630781 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630775 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-installation-pull-secrets\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.630781 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630786 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgq7\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-kube-api-access-rmgq7\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.631024 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630799 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-certificates\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.631024 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630809 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-registry-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.631024 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630817 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-bound-sa-token\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.631024 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630825 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-ca-trust-extracted\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.631024 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.630833 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c3bf732-ba93-48f8-afb6-01c2a60f5c33-trusted-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 14:59:30.778998 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.778959 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d9c8db76-blrq5"] Jul 19 14:59:30.782145 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:30.782116 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58d9c8db76-blrq5"] Jul 19 14:59:31.464498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:31.464463 2580 generic.go:356] "Generic (PLEG): container finished" podID="125a2380-9e33-47d6-926a-9af477e93234" containerID="9c37d29772d4909d7e999e1d63d3dc1a84d8cef60cc84406b2ba743be55559f8" exitCode=0 Jul 19 14:59:31.464498 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:31.464502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" event={"ID":"125a2380-9e33-47d6-926a-9af477e93234","Type":"ContainerDied","Data":"9c37d29772d4909d7e999e1d63d3dc1a84d8cef60cc84406b2ba743be55559f8"} Jul 19 14:59:31.464949 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:31.464881 2580 scope.go:117] "RemoveContainer" containerID="9c37d29772d4909d7e999e1d63d3dc1a84d8cef60cc84406b2ba743be55559f8" Jul 19 14:59:31.661363 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:31.661326 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" path="/var/lib/kubelet/pods/6c3bf732-ba93-48f8-afb6-01c2a60f5c33/volumes" Jul 19 14:59:32.268723 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:32.268690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:32.369422 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:32.369383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:32.470677 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:32.470641 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-b864s" event={"ID":"125a2380-9e33-47d6-926a-9af477e93234","Type":"ContainerStarted","Data":"559fccaf5990d08eddf0bd68ce0ecbffbb12183b1d3f0f2e08811e0c6a46618e"} Jul 19 14:59:33.973443 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:33.973324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:34.027427 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:34.027370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:36.489302 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:36.489266 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7047527-b902-4be6-94d5-9a3b377122d1" containerID="c244b2c246592fb42c54c11977a9c3ecbcd4ba39783f22d02a7ab7964c14c220" exitCode=0 Jul 19 14:59:36.489735 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:36.489338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" event={"ID":"b7047527-b902-4be6-94d5-9a3b377122d1","Type":"ContainerDied","Data":"c244b2c246592fb42c54c11977a9c3ecbcd4ba39783f22d02a7ab7964c14c220"} Jul 19 14:59:36.489735 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:36.489680 2580 scope.go:117] "RemoveContainer" containerID="c244b2c246592fb42c54c11977a9c3ecbcd4ba39783f22d02a7ab7964c14c220" Jul 19 14:59:37.494801 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:37.494765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-l6pst" event={"ID":"b7047527-b902-4be6-94d5-9a3b377122d1","Type":"ContainerStarted","Data":"6e135acc627a3d5faa7a03d56fd23a79b86bf36282dfebf0457217d98ce0444e"} Jul 19 14:59:38.986380 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:38.986335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:39.087338 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:39.087300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 14:59:39.143784 ip-10-0-140-163 kubenswrapper[2580]: I0719 14:59:39.143752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:18.379289 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:18.379248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:21.224808 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:21.224774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:21.324493 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:21.324449 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:26.305818 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.305722 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79f8844fd6-7xgrs"] Jul 19 15:00:26.306356 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.306120 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" containerName="registry" Jul 19 15:00:26.306356 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.306138 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" containerName="registry" Jul 19 15:00:26.306356 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.306243 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3bf732-ba93-48f8-afb6-01c2a60f5c33" containerName="registry" Jul 19 15:00:26.308248 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.308224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.310435 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.310408 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 19 15:00:26.310554 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.310514 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 19 15:00:26.310669 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.310644 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ldjcj\"" Jul 19 15:00:26.310811 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.310686 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 19 15:00:26.310811 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.310750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 19 15:00:26.310917 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.310846 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 19 15:00:26.314941 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.314922 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 19 15:00:26.324677 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.324654 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79f8844fd6-7xgrs"] Jul 19 15:00:26.461930 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.461890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.461945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-metrics-client-ca\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.461993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-telemeter-client-tls\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.462013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgv8\" (UniqueName: \"kubernetes.io/projected/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-kube-api-access-hjgv8\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.462041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-serving-certs-ca-bundle\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462326 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.462146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462326 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.462195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-secret-telemeter-client\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.462326 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.462255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-federate-client-tls\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.562849 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.562849 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-secret-telemeter-client\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-federate-client-tls\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-metrics-client-ca\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-telemeter-client-tls\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgv8\" (UniqueName: \"kubernetes.io/projected/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-kube-api-access-hjgv8\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.562971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-serving-certs-ca-bundle\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.563697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-metrics-client-ca\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563922 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.563835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-serving-certs-ca-bundle\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.563922 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.563904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.565609 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.565567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-secret-telemeter-client\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.565712 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.565679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-telemeter-client-tls\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.565841 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.565823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.565933 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.565918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-federate-client-tls\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.569968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.569948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgv8\" (UniqueName: \"kubernetes.io/projected/24af6a87-1591-48d5-9f1e-ab6f6f6634a4-kube-api-access-hjgv8\") pod \"telemeter-client-79f8844fd6-7xgrs\" (UID: \"24af6a87-1591-48d5-9f1e-ab6f6f6634a4\") " pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.617512 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.617478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" Jul 19 15:00:26.749049 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:26.749006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79f8844fd6-7xgrs"] Jul 19 15:00:26.751243 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:00:26.751218 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af6a87_1591_48d5_9f1e_ab6f6f6634a4.slice/crio-eadb5c32f07f615c712ffe7dce811bd25eabaf0ddce02ecc9c7fdc456680bc8b WatchSource:0}: Error finding container eadb5c32f07f615c712ffe7dce811bd25eabaf0ddce02ecc9c7fdc456680bc8b: Status 404 returned error can't find the container with id eadb5c32f07f615c712ffe7dce811bd25eabaf0ddce02ecc9c7fdc456680bc8b Jul 19 15:00:27.677368 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:27.677331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" event={"ID":"24af6a87-1591-48d5-9f1e-ab6f6f6634a4","Type":"ContainerStarted","Data":"eadb5c32f07f615c712ffe7dce811bd25eabaf0ddce02ecc9c7fdc456680bc8b"} Jul 19 15:00:28.682894 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:28.682860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" event={"ID":"24af6a87-1591-48d5-9f1e-ab6f6f6634a4","Type":"ContainerStarted","Data":"0e276fa0e7e7d7c24e88c0ccb49ab7be98b35e81532784ae4b2d12b297c764ca"} Jul 19 15:00:30.692809 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:30.692772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" event={"ID":"24af6a87-1591-48d5-9f1e-ab6f6f6634a4","Type":"ContainerStarted","Data":"e87e7642861366030774002bb61a4bae907950f88f576650e2bcfe34136e7abb"} Jul 19 15:00:30.693203 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:30.692836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" event={"ID":"24af6a87-1591-48d5-9f1e-ab6f6f6634a4","Type":"ContainerStarted","Data":"2dd3f9adfe5cbef56e8d98cb7340e27628fcccbef9ea829a8c56e5fb0c6473af"} Jul 19 15:00:30.713137 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:30.713081 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79f8844fd6-7xgrs" podStartSLOduration=1.7891421140000001 podStartE2EDuration="4.713059258s" totalImagesPullingTime="2.923917144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:00:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:00:30.712375938 +0000 UTC m=+181.576650735" watchObservedRunningTime="2026-07-19 15:00:30.713059258 +0000 UTC m=+181.577334060" Jul 19 15:00:31.481087 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:31.481046 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:31.549404 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:31.549371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:42.719010 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:42.718976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:42.844505 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:42.844469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:42.897197 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:42.897163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:52.662729 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:52.662689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:00:52.711852 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:00:52.711820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:21.108517 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:21.108476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:36.391803 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:36.391753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:36.438662 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:36.438617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:39.758025 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:39.757993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:48.319336 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:48.319295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:48.371796 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:48.371756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:50.990541 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:50.990479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:51.042491 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:51.042457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:01:51.092401 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:01:51.092361 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:02:03.978989 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:03.978903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:02:04.076473 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:04.076416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:02:29.552500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:29.552467 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:02:29.557536 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:29.557511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:02:29.558872 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:29.558853 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:02:29.563873 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:29.563854 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 19 15:02:29.564510 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:29.564494 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:02:39.736646 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:39.736602 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:02:40.316637 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.316600 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd"] Jul 19 15:02:40.319780 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.319765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.322029 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.321994 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 19 15:02:40.322730 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.322683 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f628c\"" Jul 19 15:02:40.322842 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.322731 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 19 15:02:40.328222 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.328197 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd"] Jul 19 15:02:40.375932 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.375899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt6r\" (UniqueName: \"kubernetes.io/projected/5009e2a8-9251-4f59-8e72-c355191491c0-kube-api-access-nmt6r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.376088 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.375946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.376088 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.375999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.477261 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.477223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.477431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.477290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt6r\" (UniqueName: \"kubernetes.io/projected/5009e2a8-9251-4f59-8e72-c355191491c0-kube-api-access-nmt6r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.477431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.477320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.477659 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.477640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.477702 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.477676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.486141 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.486119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt6r\" (UniqueName: \"kubernetes.io/projected/5009e2a8-9251-4f59-8e72-c355191491c0-kube-api-access-nmt6r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.629833 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.629801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:40.754570 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.754540 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd"] Jul 19 15:02:40.757240 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:02:40.757214 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5009e2a8_9251_4f59_8e72_c355191491c0.slice/crio-f1b34aa242cdcf54726a83ed866d16b1247538e21231d46089138db49170aa53 WatchSource:0}: Error finding container f1b34aa242cdcf54726a83ed866d16b1247538e21231d46089138db49170aa53: Status 404 returned error can't find the container with id f1b34aa242cdcf54726a83ed866d16b1247538e21231d46089138db49170aa53 Jul 19 15:02:40.759214 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.759197 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:02:40.941023 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:40.940943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:02:41.166967 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:41.166931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" event={"ID":"5009e2a8-9251-4f59-8e72-c355191491c0","Type":"ContainerStarted","Data":"f1b34aa242cdcf54726a83ed866d16b1247538e21231d46089138db49170aa53"} Jul 19 15:02:47.190979 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:47.190947 2580 generic.go:356] "Generic (PLEG): container finished" podID="5009e2a8-9251-4f59-8e72-c355191491c0" containerID="ef63f38b2ef8ac56bd52d812c4d0fb64f13a8537f20fd00bece9ab8c3685cd9b" exitCode=0 Jul 19 15:02:47.191286 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:47.191021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" event={"ID":"5009e2a8-9251-4f59-8e72-c355191491c0","Type":"ContainerDied","Data":"ef63f38b2ef8ac56bd52d812c4d0fb64f13a8537f20fd00bece9ab8c3685cd9b"} Jul 19 15:02:50.204816 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:50.204780 2580 generic.go:356] "Generic (PLEG): container finished" podID="5009e2a8-9251-4f59-8e72-c355191491c0" containerID="7bafaaed46272dcf42d56bd1635ccd8c40bad2ad88b28c9848d385f986e8ad04" exitCode=0 Jul 19 15:02:50.205192 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:50.204848 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" event={"ID":"5009e2a8-9251-4f59-8e72-c355191491c0","Type":"ContainerDied","Data":"7bafaaed46272dcf42d56bd1635ccd8c40bad2ad88b28c9848d385f986e8ad04"} Jul 19 15:02:57.236161 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:57.236129 2580 generic.go:356] "Generic (PLEG): container finished" podID="5009e2a8-9251-4f59-8e72-c355191491c0" containerID="dd664881cb76e7990dd4816fd7c9a5b50711446a3ee1a177fb72f634061de1be" exitCode=0 Jul 19 15:02:57.236161 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:57.236170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" event={"ID":"5009e2a8-9251-4f59-8e72-c355191491c0","Type":"ContainerDied","Data":"dd664881cb76e7990dd4816fd7c9a5b50711446a3ee1a177fb72f634061de1be"} Jul 19 15:02:58.365926 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.365902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:58.419922 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.419871 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-bundle\") pod \"5009e2a8-9251-4f59-8e72-c355191491c0\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " Jul 19 15:02:58.421018 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.420936 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-bundle" (OuterVolumeSpecName: "bundle") pod "5009e2a8-9251-4f59-8e72-c355191491c0" (UID: "5009e2a8-9251-4f59-8e72-c355191491c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:02:58.422200 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.421177 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmt6r\" (UniqueName: \"kubernetes.io/projected/5009e2a8-9251-4f59-8e72-c355191491c0-kube-api-access-nmt6r\") pod \"5009e2a8-9251-4f59-8e72-c355191491c0\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " Jul 19 15:02:58.422200 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.421275 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-util\") pod \"5009e2a8-9251-4f59-8e72-c355191491c0\" (UID: \"5009e2a8-9251-4f59-8e72-c355191491c0\") " Jul 19 15:02:58.422200 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.421668 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:02:58.426497 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.426466 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5009e2a8-9251-4f59-8e72-c355191491c0-kube-api-access-nmt6r" (OuterVolumeSpecName: "kube-api-access-nmt6r") pod "5009e2a8-9251-4f59-8e72-c355191491c0" (UID: "5009e2a8-9251-4f59-8e72-c355191491c0"). InnerVolumeSpecName "kube-api-access-nmt6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:02:58.427760 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.427639 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-util" (OuterVolumeSpecName: "util") pod "5009e2a8-9251-4f59-8e72-c355191491c0" (UID: "5009e2a8-9251-4f59-8e72-c355191491c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:02:58.522177 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.522090 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmt6r\" (UniqueName: \"kubernetes.io/projected/5009e2a8-9251-4f59-8e72-c355191491c0-kube-api-access-nmt6r\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:02:58.522177 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:58.522125 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5009e2a8-9251-4f59-8e72-c355191491c0-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:02:59.246068 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:59.246040 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" Jul 19 15:02:59.246234 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:59.246031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540m2tkd" event={"ID":"5009e2a8-9251-4f59-8e72-c355191491c0","Type":"ContainerDied","Data":"f1b34aa242cdcf54726a83ed866d16b1247538e21231d46089138db49170aa53"} Jul 19 15:02:59.246234 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:02:59.246162 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b34aa242cdcf54726a83ed866d16b1247538e21231d46089138db49170aa53" Jul 19 15:03:01.083977 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:01.083940 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:02.449903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:02.449866 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:03.015214 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015178 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf"] Jul 19 15:03:03.015555 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015528 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="util" Jul 19 15:03:03.015555 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015551 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="util" Jul 19 15:03:03.015727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015569 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="pull" Jul 19 15:03:03.015727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015574 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="pull" Jul 19 15:03:03.015727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015597 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="extract" Jul 19 15:03:03.015727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015607 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="extract" Jul 19 15:03:03.015727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.015667 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5009e2a8-9251-4f59-8e72-c355191491c0" containerName="extract" Jul 19 15:03:03.018729 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.018710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.020949 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.020924 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 19 15:03:03.020949 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.020941 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 19 15:03:03.021112 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.021059 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qks7c\"" Jul 19 15:03:03.033884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.033856 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf"] Jul 19 15:03:03.161261 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.161224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmcq\" (UniqueName: \"kubernetes.io/projected/84048e4d-45ef-439a-a771-816780258102-kube-api-access-7zmcq\") pod \"cert-manager-operator-controller-manager-55c444bf86-wnqtf\" (UID: \"84048e4d-45ef-439a-a771-816780258102\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.161445 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.161291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84048e4d-45ef-439a-a771-816780258102-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wnqtf\" (UID: \"84048e4d-45ef-439a-a771-816780258102\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.261831 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.261800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmcq\" (UniqueName: \"kubernetes.io/projected/84048e4d-45ef-439a-a771-816780258102-kube-api-access-7zmcq\") pod \"cert-manager-operator-controller-manager-55c444bf86-wnqtf\" (UID: \"84048e4d-45ef-439a-a771-816780258102\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.261994 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.261869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84048e4d-45ef-439a-a771-816780258102-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wnqtf\" (UID: \"84048e4d-45ef-439a-a771-816780258102\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.262294 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.262275 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84048e4d-45ef-439a-a771-816780258102-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wnqtf\" (UID: \"84048e4d-45ef-439a-a771-816780258102\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.272113 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.270808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmcq\" (UniqueName: \"kubernetes.io/projected/84048e4d-45ef-439a-a771-816780258102-kube-api-access-7zmcq\") pod \"cert-manager-operator-controller-manager-55c444bf86-wnqtf\" (UID: \"84048e4d-45ef-439a-a771-816780258102\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.327532 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.327495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" Jul 19 15:03:03.454183 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:03.454156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf"] Jul 19 15:03:03.456542 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:03:03.456519 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84048e4d_45ef_439a_a771_816780258102.slice/crio-94a8fac380bd4da57b931d084fbab2881011dfdc3351761c0eda32be1949e869 WatchSource:0}: Error finding container 94a8fac380bd4da57b931d084fbab2881011dfdc3351761c0eda32be1949e869: Status 404 returned error can't find the container with id 94a8fac380bd4da57b931d084fbab2881011dfdc3351761c0eda32be1949e869 Jul 19 15:03:04.268903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:04.268866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" event={"ID":"84048e4d-45ef-439a-a771-816780258102","Type":"ContainerStarted","Data":"94a8fac380bd4da57b931d084fbab2881011dfdc3351761c0eda32be1949e869"} Jul 19 15:03:04.712192 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:04.712142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:04.811197 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:04.811156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:05.275086 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:05.275051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" event={"ID":"84048e4d-45ef-439a-a771-816780258102","Type":"ContainerStarted","Data":"e37136dcf6c3cbe0afe44870602bf5d4278eea108ad12b6efc372fd8d9feaf24"} Jul 19 15:03:05.295520 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:05.295453 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wnqtf" podStartSLOduration=1.6804468689999998 podStartE2EDuration="3.295434587s" totalImagesPullingTime="1.614987718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:05.293668485 +0000 UTC m=+336.157943281" watchObservedRunningTime="2026-07-19 15:03:05.295434587 +0000 UTC m=+336.159709381" Jul 19 15:03:05.762278 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:05.762239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:05.876367 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:05.876323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:06.724916 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.724883 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl"] Jul 19 15:03:06.728423 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.728399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.730847 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.730828 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 19 15:03:06.730968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.730854 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 19 15:03:06.730968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.730897 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f628c\"" Jul 19 15:03:06.736240 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.736211 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl"] Jul 19 15:03:06.789482 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.789438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.789914 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.789486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvm9\" (UniqueName: \"kubernetes.io/projected/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-kube-api-access-pmvm9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.789914 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.789542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.890910 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.890870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.891098 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.890976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.891098 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.891005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvm9\" (UniqueName: \"kubernetes.io/projected/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-kube-api-access-pmvm9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.891320 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.891296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.891387 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.891324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.898492 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.898452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvm9\" (UniqueName: \"kubernetes.io/projected/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-kube-api-access-pmvm9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:06.911863 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:06.911802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:07.013118 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:07.013034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:07.038670 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:07.038644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:07.168738 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:07.168671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl"] Jul 19 15:03:07.170805 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:03:07.170775 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8041b115_dc81_4ad9_9ba9_642c1a3c81fc.slice/crio-2cf30da6b06b92869d959502493763e76f100c4577b50e3bf8a7139f76d541a2 WatchSource:0}: Error finding container 2cf30da6b06b92869d959502493763e76f100c4577b50e3bf8a7139f76d541a2: Status 404 returned error can't find the container with id 2cf30da6b06b92869d959502493763e76f100c4577b50e3bf8a7139f76d541a2 Jul 19 15:03:07.285651 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:07.285622 2580 generic.go:356] "Generic (PLEG): container finished" podID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerID="1d13ff1fc53847b7aa989d82440a6c321ab3991653e36528dcd0af1c940625f5" exitCode=0 Jul 19 15:03:07.285777 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:07.285696 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" event={"ID":"8041b115-dc81-4ad9-9ba9-642c1a3c81fc","Type":"ContainerDied","Data":"1d13ff1fc53847b7aa989d82440a6c321ab3991653e36528dcd0af1c940625f5"} Jul 19 15:03:07.285777 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:07.285733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" event={"ID":"8041b115-dc81-4ad9-9ba9-642c1a3c81fc","Type":"ContainerStarted","Data":"2cf30da6b06b92869d959502493763e76f100c4577b50e3bf8a7139f76d541a2"} Jul 19 15:03:09.115201 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:09.115157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:09.310288 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:09.310250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:11.689812 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:11.689767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:11.790121 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:11.790089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:13.772851 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:13.772814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:13.823162 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:13.823128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:17.323867 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.323837 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fzwd7"] Jul 19 15:03:17.324677 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.324647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:17.330001 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.329971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.332535 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.332513 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 19 15:03:17.332826 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.332557 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pkbzb\"" Jul 19 15:03:17.333312 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.333295 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 19 15:03:17.336614 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.336571 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fzwd7"] Jul 19 15:03:17.487720 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.487687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhsb\" (UniqueName: \"kubernetes.io/projected/ed62aced-d822-4c1e-8d50-525c08377949-kube-api-access-bzhsb\") pod \"cert-manager-55b4d954d5-fzwd7\" (UID: \"ed62aced-d822-4c1e-8d50-525c08377949\") " pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.487720 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.487726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed62aced-d822-4c1e-8d50-525c08377949-bound-sa-token\") pod \"cert-manager-55b4d954d5-fzwd7\" (UID: \"ed62aced-d822-4c1e-8d50-525c08377949\") " pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.588888 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.588796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsb\" (UniqueName: \"kubernetes.io/projected/ed62aced-d822-4c1e-8d50-525c08377949-kube-api-access-bzhsb\") pod \"cert-manager-55b4d954d5-fzwd7\" (UID: \"ed62aced-d822-4c1e-8d50-525c08377949\") " pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.588888 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.588843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed62aced-d822-4c1e-8d50-525c08377949-bound-sa-token\") pod \"cert-manager-55b4d954d5-fzwd7\" (UID: \"ed62aced-d822-4c1e-8d50-525c08377949\") " pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.599764 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.599731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed62aced-d822-4c1e-8d50-525c08377949-bound-sa-token\") pod \"cert-manager-55b4d954d5-fzwd7\" (UID: \"ed62aced-d822-4c1e-8d50-525c08377949\") " pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.599903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.599842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhsb\" (UniqueName: \"kubernetes.io/projected/ed62aced-d822-4c1e-8d50-525c08377949-kube-api-access-bzhsb\") pod \"cert-manager-55b4d954d5-fzwd7\" (UID: \"ed62aced-d822-4c1e-8d50-525c08377949\") " pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.641161 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.641134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fzwd7" Jul 19 15:03:17.789647 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:17.789608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fzwd7"] Jul 19 15:03:17.792664 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:03:17.792641 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded62aced_d822_4c1e_8d50_525c08377949.slice/crio-fef0576ef025ed88b45cf981021308c60211e8486c0b3611f233ba66791a2303 WatchSource:0}: Error finding container fef0576ef025ed88b45cf981021308c60211e8486c0b3611f233ba66791a2303: Status 404 returned error can't find the container with id fef0576ef025ed88b45cf981021308c60211e8486c0b3611f233ba66791a2303 Jul 19 15:03:18.337288 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:18.337250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fzwd7" event={"ID":"ed62aced-d822-4c1e-8d50-525c08377949","Type":"ContainerStarted","Data":"fef0576ef025ed88b45cf981021308c60211e8486c0b3611f233ba66791a2303"} Jul 19 15:03:21.350801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:21.350768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fzwd7" event={"ID":"ed62aced-d822-4c1e-8d50-525c08377949","Type":"ContainerStarted","Data":"b6948ebb29d1928996ecc95801990436bc75c09c8bdf6011680c4bb2c2f685da"} Jul 19 15:03:21.365297 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:21.365241 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fzwd7" podStartSLOduration=1.862805728 podStartE2EDuration="4.365221831s" totalImagesPullingTime="2.502416103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:21.36384841 +0000 UTC m=+352.228123206" watchObservedRunningTime="2026-07-19 15:03:21.365221831 +0000 UTC m=+352.229496628" Jul 19 15:03:28.384185 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:28.384148 2580 generic.go:356] "Generic (PLEG): container finished" podID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerID="da9503cbf02ad111b4b4e987f1c8ddd098a367c84b74c874201cdf3fd6bff780" exitCode=0 Jul 19 15:03:28.384528 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:28.384225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" event={"ID":"8041b115-dc81-4ad9-9ba9-642c1a3c81fc","Type":"ContainerDied","Data":"da9503cbf02ad111b4b4e987f1c8ddd098a367c84b74c874201cdf3fd6bff780"} Jul 19 15:03:28.540560 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:03:28.540534 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8041b115_dc81_4ad9_9ba9_642c1a3c81fc.slice/crio-4f5d21532e6263925e05d2e7bf7336e4c2706a04205d49164241838d1a2a3000.scope\": RecentStats: unable to find data in memory cache]" Jul 19 15:03:29.391171 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:29.391137 2580 generic.go:356] "Generic (PLEG): container finished" podID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerID="4f5d21532e6263925e05d2e7bf7336e4c2706a04205d49164241838d1a2a3000" exitCode=0 Jul 19 15:03:29.391539 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:29.391200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" event={"ID":"8041b115-dc81-4ad9-9ba9-642c1a3c81fc","Type":"ContainerDied","Data":"4f5d21532e6263925e05d2e7bf7336e4c2706a04205d49164241838d1a2a3000"} Jul 19 15:03:30.540670 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.540635 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:30.587334 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.587300 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-util\") pod \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " Jul 19 15:03:30.587510 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.587374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmvm9\" (UniqueName: \"kubernetes.io/projected/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-kube-api-access-pmvm9\") pod \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " Jul 19 15:03:30.587510 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.587416 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-bundle\") pod \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\" (UID: \"8041b115-dc81-4ad9-9ba9-642c1a3c81fc\") " Jul 19 15:03:30.587902 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.587874 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-bundle" (OuterVolumeSpecName: "bundle") pod "8041b115-dc81-4ad9-9ba9-642c1a3c81fc" (UID: "8041b115-dc81-4ad9-9ba9-642c1a3c81fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:30.589432 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.589405 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-kube-api-access-pmvm9" (OuterVolumeSpecName: "kube-api-access-pmvm9") pod "8041b115-dc81-4ad9-9ba9-642c1a3c81fc" (UID: "8041b115-dc81-4ad9-9ba9-642c1a3c81fc"). InnerVolumeSpecName "kube-api-access-pmvm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:03:30.591856 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.591830 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-util" (OuterVolumeSpecName: "util") pod "8041b115-dc81-4ad9-9ba9-642c1a3c81fc" (UID: "8041b115-dc81-4ad9-9ba9-642c1a3c81fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:30.688302 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.688215 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:03:30.688302 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.688245 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:03:30.688302 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:30.688254 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvm9\" (UniqueName: \"kubernetes.io/projected/8041b115-dc81-4ad9-9ba9-642c1a3c81fc-kube-api-access-pmvm9\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:03:31.401880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:31.401853 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" Jul 19 15:03:31.402091 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:31.401854 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fc4fvl" event={"ID":"8041b115-dc81-4ad9-9ba9-642c1a3c81fc","Type":"ContainerDied","Data":"2cf30da6b06b92869d959502493763e76f100c4577b50e3bf8a7139f76d541a2"} Jul 19 15:03:31.402091 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:31.401974 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf30da6b06b92869d959502493763e76f100c4577b50e3bf8a7139f76d541a2" Jul 19 15:03:34.239550 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:34.239510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:34.333111 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:34.333079 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:39.746377 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:39.746332 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:39.848745 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:39.848715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:41.456825 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.456783 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp"] Jul 19 15:03:41.457167 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457138 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="extract" Jul 19 15:03:41.457167 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457149 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="extract" Jul 19 15:03:41.457167 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457165 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="util" Jul 19 15:03:41.457276 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457171 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="util" Jul 19 15:03:41.457276 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457178 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="pull" Jul 19 15:03:41.457276 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457183 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="pull" Jul 19 15:03:41.457276 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.457229 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8041b115-dc81-4ad9-9ba9-642c1a3c81fc" containerName="extract" Jul 19 15:03:41.463298 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.463278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.465884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.465862 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 19 15:03:41.466005 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.465902 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 19 15:03:41.466005 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.465961 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f628c\"" Jul 19 15:03:41.466807 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.466786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp"] Jul 19 15:03:41.573628 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.573578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.573796 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.573643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dzq\" (UniqueName: \"kubernetes.io/projected/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-kube-api-access-r7dzq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.573796 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.573668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.674597 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.674548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.674777 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.674604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dzq\" (UniqueName: \"kubernetes.io/projected/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-kube-api-access-r7dzq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.674777 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.674653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.674920 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.674900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.674983 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.674951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.682353 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.682326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dzq\" (UniqueName: \"kubernetes.io/projected/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-kube-api-access-r7dzq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.773004 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.772929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:41.898079 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.898052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp"] Jul 19 15:03:41.900251 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:03:41.900223 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb5a2fa_a3ba_43bf_8759_7e90a15a23e9.slice/crio-fd6e2fce75c1bbcfd779a18e450ec14c6b7eb0328f6f6c07117bb3d40b07e1ea WatchSource:0}: Error finding container fd6e2fce75c1bbcfd779a18e450ec14c6b7eb0328f6f6c07117bb3d40b07e1ea: Status 404 returned error can't find the container with id fd6e2fce75c1bbcfd779a18e450ec14c6b7eb0328f6f6c07117bb3d40b07e1ea Jul 19 15:03:41.964955 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:41.964926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:42.454223 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:42.454190 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerID="64d535fbfb8d33b928a923a6289d497e7b9a0f9401bb56c688d0bec31a75957a" exitCode=0 Jul 19 15:03:42.454424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:42.454265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" event={"ID":"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9","Type":"ContainerDied","Data":"64d535fbfb8d33b928a923a6289d497e7b9a0f9401bb56c688d0bec31a75957a"} Jul 19 15:03:42.454424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:42.454322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" event={"ID":"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9","Type":"ContainerStarted","Data":"fd6e2fce75c1bbcfd779a18e450ec14c6b7eb0328f6f6c07117bb3d40b07e1ea"} Jul 19 15:03:43.460279 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:43.460192 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerID="f63ddc43b1104e0359bb4d32b0657454e30521e462d6843fd98008d6f64a3203" exitCode=0 Jul 19 15:03:43.460713 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:43.460282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" event={"ID":"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9","Type":"ContainerDied","Data":"f63ddc43b1104e0359bb4d32b0657454e30521e462d6843fd98008d6f64a3203"} Jul 19 15:03:44.467071 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:44.467038 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerID="dcb68076c67bd85719c864658f26293834a199f9976eb0148b7f4b9085567519" exitCode=0 Jul 19 15:03:44.467501 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:44.467098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" event={"ID":"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9","Type":"ContainerDied","Data":"dcb68076c67bd85719c864658f26293834a199f9976eb0148b7f4b9085567519"} Jul 19 15:03:45.595430 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.595408 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:45.705699 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.705665 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-util\") pod \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " Jul 19 15:03:45.705898 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.705716 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7dzq\" (UniqueName: \"kubernetes.io/projected/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-kube-api-access-r7dzq\") pod \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " Jul 19 15:03:45.705898 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.705748 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-bundle\") pod \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\" (UID: \"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9\") " Jul 19 15:03:45.706470 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.706447 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-bundle" (OuterVolumeSpecName: "bundle") pod "9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" (UID: "9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:45.707815 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.707792 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-kube-api-access-r7dzq" (OuterVolumeSpecName: "kube-api-access-r7dzq") pod "9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" (UID: "9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9"). InnerVolumeSpecName "kube-api-access-r7dzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:03:45.711620 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.711594 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-util" (OuterVolumeSpecName: "util") pod "9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" (UID: "9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:03:45.807108 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.807006 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:03:45.807108 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.807052 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dzq\" (UniqueName: \"kubernetes.io/projected/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-kube-api-access-r7dzq\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:03:45.807108 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:45.807066 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:03:46.198973 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:46.198937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:46.265616 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:46.265567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:46.478130 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:46.478054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" event={"ID":"9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9","Type":"ContainerDied","Data":"fd6e2fce75c1bbcfd779a18e450ec14c6b7eb0328f6f6c07117bb3d40b07e1ea"} Jul 19 15:03:46.478266 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:46.478135 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd6e2fce75c1bbcfd779a18e450ec14c6b7eb0328f6f6c07117bb3d40b07e1ea" Jul 19 15:03:46.478266 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:46.478067 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqmlsp" Jul 19 15:03:46.998625 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:46.998577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:47.051390 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:47.051358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:57.610843 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:57.610802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:57.716237 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:57.716207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:58.400311 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.400277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:58.644353 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.644320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:58.752436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.752349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:58.760292 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760263 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt"] Jul 19 15:03:58.760618 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760592 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="extract" Jul 19 15:03:58.760618 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760609 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="extract" Jul 19 15:03:58.760801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760627 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="pull" Jul 19 15:03:58.760801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760634 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="pull" Jul 19 15:03:58.760801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760642 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="util" Jul 19 15:03:58.760801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760647 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="util" Jul 19 15:03:58.760801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.760705 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb5a2fa-a3ba-43bf-8759-7e90a15a23e9" containerName="extract" Jul 19 15:03:58.762648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.762633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.765178 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.765158 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 19 15:03:58.766217 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.766197 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 19 15:03:58.766528 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.766508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vwrkx\"" Jul 19 15:03:58.766641 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.766605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 19 15:03:58.766925 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.766910 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 19 15:03:58.780382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.780353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt"] Jul 19 15:03:58.812382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.812343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mn5p\" (UniqueName: \"kubernetes.io/projected/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-kube-api-access-5mn5p\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.812543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.812394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.812543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.812456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.812543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.812485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.812543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.812507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.913182 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.913148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mn5p\" (UniqueName: \"kubernetes.io/projected/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-kube-api-access-5mn5p\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.913365 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.913214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.913365 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.913244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.913365 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.913268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.913365 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.913306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.913749 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.913722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.914087 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.914057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:58.916424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.916398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.916524 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.916445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.916682 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.916662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:58.925758 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:58.925733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mn5p\" (UniqueName: \"kubernetes.io/projected/af59f2f3-37b4-4c1a-8c05-ed811336b1ff-kube-api-access-5mn5p\") pod \"opendatahub-operator-controller-manager-74d6d868c8-x6vbt\" (UID: \"af59f2f3-37b4-4c1a-8c05-ed811336b1ff\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:59.014722 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.014631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:59.073110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.073081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:03:59.115956 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.115922 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:59.206908 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.206860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt"] Jul 19 15:03:59.208996 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.208964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:59.209979 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:03:59.209957 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf59f2f3_37b4_4c1a_8c05_ed811336b1ff.slice/crio-7b6b52f09cd76548f005342d6f80d3192245b139922c247c144662fce4f02180 WatchSource:0}: Error finding container 7b6b52f09cd76548f005342d6f80d3192245b139922c247c144662fce4f02180: Status 404 returned error can't find the container with id 7b6b52f09cd76548f005342d6f80d3192245b139922c247c144662fce4f02180 Jul 19 15:03:59.276320 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.276227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:59.377021 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.376982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:59.482805 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.482773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:03:59.540926 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.540843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" event={"ID":"af59f2f3-37b4-4c1a-8c05-ed811336b1ff","Type":"ContainerStarted","Data":"7b6b52f09cd76548f005342d6f80d3192245b139922c247c144662fce4f02180"} Jul 19 15:03:59.582663 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:03:59.582628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:02.558525 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:02.558442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" event={"ID":"af59f2f3-37b4-4c1a-8c05-ed811336b1ff","Type":"ContainerStarted","Data":"d1411c588dfed5b9985a2378766d6df858a59e12d780cfb110f12a65d7ab20b0"} Jul 19 15:04:03.563732 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:03.563694 2580 generic.go:356] "Generic (PLEG): container finished" podID="af59f2f3-37b4-4c1a-8c05-ed811336b1ff" containerID="d1411c588dfed5b9985a2378766d6df858a59e12d780cfb110f12a65d7ab20b0" exitCode=0 Jul 19 15:04:03.564198 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:03.563751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" event={"ID":"af59f2f3-37b4-4c1a-8c05-ed811336b1ff","Type":"ContainerDied","Data":"d1411c588dfed5b9985a2378766d6df858a59e12d780cfb110f12a65d7ab20b0"} Jul 19 15:04:04.570734 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:04.570688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" event={"ID":"af59f2f3-37b4-4c1a-8c05-ed811336b1ff","Type":"ContainerStarted","Data":"7f96e7cf467488d986c36adec97702deb5c608541e4aa34e3c8ff118c36a47ea"} Jul 19 15:04:04.571187 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:04.570775 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:04:04.590637 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:04.590562 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" podStartSLOduration=2.18319085 podStartE2EDuration="6.590547025s" totalImagesPullingTime="3.321254164s" totalInitContainerRuntime="1.086102011s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:04.589821875 +0000 UTC m=+395.454096671" watchObservedRunningTime="2026-07-19 15:04:04.590547025 +0000 UTC m=+395.454821821" Jul 19 15:04:10.860465 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:10.860425 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:10.960009 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:10.959969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:14.332179 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:14.332147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:14.436220 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:14.436177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:14.761464 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:14.761432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:15.577753 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:15.577717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-x6vbt" Jul 19 15:04:17.707978 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:17.707936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:17.848179 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:17.848138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:18.265605 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.265549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:18.413622 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.413574 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-647d897b75-94xbq"] Jul 19 15:04:18.421901 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.421874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.425247 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.425227 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 19 15:04:18.425373 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.425321 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 19 15:04:18.433879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.433857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-647d897b75-94xbq"] Jul 19 15:04:18.446298 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.446275 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ntjhh\"" Jul 19 15:04:18.540814 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.540692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:18.577542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.577506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-tls-certs\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.577721 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.577552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-tmp\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.577721 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.577636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlcwq\" (UniqueName: \"kubernetes.io/projected/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-kube-api-access-qlcwq\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.610957 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.610920 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rp424"] Jul 19 15:04:18.614346 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.614328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.616564 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.616544 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 19 15:04:18.616667 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.616596 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fm6j4\"" Jul 19 15:04:18.621904 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.621879 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rp424"] Jul 19 15:04:18.678421 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.678380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-tmp\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.678621 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.678486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlcwq\" (UniqueName: \"kubernetes.io/projected/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-kube-api-access-qlcwq\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.678621 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.678562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-tls-certs\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.681163 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.681126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-tmp\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.681354 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.681330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-tls-certs\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.686528 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.686497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlcwq\" (UniqueName: \"kubernetes.io/projected/76cfdf0d-eef2-4908-bf8f-3403c4aec4b7-kube-api-access-qlcwq\") pod \"kube-auth-proxy-647d897b75-94xbq\" (UID: \"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7\") " pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.690175 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.690147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:18.731164 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.731133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" Jul 19 15:04:18.779898 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.779857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8phkl\" (UniqueName: \"kubernetes.io/projected/0c8f1435-8393-46f7-b8df-7a696a2863d6-kube-api-access-8phkl\") pod \"odh-model-controller-7998c67dc4-rp424\" (UID: \"0c8f1435-8393-46f7-b8df-7a696a2863d6\") " pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.780048 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.780018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c8f1435-8393-46f7-b8df-7a696a2863d6-cert\") pod \"odh-model-controller-7998c67dc4-rp424\" (UID: \"0c8f1435-8393-46f7-b8df-7a696a2863d6\") " pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.787478 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.787447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:18.864784 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.864759 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-647d897b75-94xbq"] Jul 19 15:04:18.867487 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:18.867460 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cfdf0d_eef2_4908_bf8f_3403c4aec4b7.slice/crio-56fa08c57f3e9f4b1d4d7d264b67a08de4e692f01bd40487439cf3f1f814c54e WatchSource:0}: Error finding container 56fa08c57f3e9f4b1d4d7d264b67a08de4e692f01bd40487439cf3f1f814c54e: Status 404 returned error can't find the container with id 56fa08c57f3e9f4b1d4d7d264b67a08de4e692f01bd40487439cf3f1f814c54e Jul 19 15:04:18.881431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.881402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8phkl\" (UniqueName: \"kubernetes.io/projected/0c8f1435-8393-46f7-b8df-7a696a2863d6-kube-api-access-8phkl\") pod \"odh-model-controller-7998c67dc4-rp424\" (UID: \"0c8f1435-8393-46f7-b8df-7a696a2863d6\") " pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.881556 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.881463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c8f1435-8393-46f7-b8df-7a696a2863d6-cert\") pod \"odh-model-controller-7998c67dc4-rp424\" (UID: \"0c8f1435-8393-46f7-b8df-7a696a2863d6\") " pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.883850 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.883828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c8f1435-8393-46f7-b8df-7a696a2863d6-cert\") pod \"odh-model-controller-7998c67dc4-rp424\" (UID: \"0c8f1435-8393-46f7-b8df-7a696a2863d6\") " pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.888997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.888979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8phkl\" (UniqueName: \"kubernetes.io/projected/0c8f1435-8393-46f7-b8df-7a696a2863d6-kube-api-access-8phkl\") pod \"odh-model-controller-7998c67dc4-rp424\" (UID: \"0c8f1435-8393-46f7-b8df-7a696a2863d6\") " pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:18.896967 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.896938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:18.924790 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:18.924762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:19.079833 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.079810 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rp424"] Jul 19 15:04:19.081468 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:19.081443 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8f1435_8393_46f7_b8df_7a696a2863d6.slice/crio-b3f5d37b06951ac50cc04d0b3bd2264c6b10c685d272b68dd98ffee3292f6b21 WatchSource:0}: Error finding container b3f5d37b06951ac50cc04d0b3bd2264c6b10c685d272b68dd98ffee3292f6b21: Status 404 returned error can't find the container with id b3f5d37b06951ac50cc04d0b3bd2264c6b10c685d272b68dd98ffee3292f6b21 Jul 19 15:04:19.452351 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.452316 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zmckx"] Jul 19 15:04:19.456125 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.456101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.456718 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.456666 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:19.458429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.458213 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 19 15:04:19.458429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.458220 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 19 15:04:19.458429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.458261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 19 15:04:19.458429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.458308 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 15:04:19.458790 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.458563 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-447ml\"" Jul 19 15:04:19.465794 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.465767 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zmckx"] Jul 19 15:04:19.555685 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.555648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:19.588145 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-local-certs\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.588333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-cacerts\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.588333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-token\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.588333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.588333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.588565 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.588565 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.588374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772qn\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-kube-api-access-772qn\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.589297 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.589273 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-57bzz"] Jul 19 15:04:19.592849 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.592822 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:19.594795 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.594759 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 19 15:04:19.594907 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.594870 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-cr6z6\"" Jul 19 15:04:19.602103 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.602041 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-57bzz"] Jul 19 15:04:19.621613 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.621559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:19.644466 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.644430 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" event={"ID":"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7","Type":"ContainerStarted","Data":"56fa08c57f3e9f4b1d4d7d264b67a08de4e692f01bd40487439cf3f1f814c54e"} Jul 19 15:04:19.646184 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.646120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" event={"ID":"0c8f1435-8393-46f7-b8df-7a696a2863d6","Type":"ContainerStarted","Data":"b3f5d37b06951ac50cc04d0b3bd2264c6b10c685d272b68dd98ffee3292f6b21"} Jul 19 15:04:19.689924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-token\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.689924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.689924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.689924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.689924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772qn\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-kube-api-access-772qn\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.690323 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mrc\" (UniqueName: \"kubernetes.io/projected/a3210330-e30d-4248-8a9f-5dc79531d1ef-kube-api-access-g5mrc\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:19.690323 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.689978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3210330-e30d-4248-8a9f-5dc79531d1ef-tls-certs\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:19.690323 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.690035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-local-certs\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.690323 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.690062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-cacerts\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.690642 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.690618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.693534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.693505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-cacerts\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.694012 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.693989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.696135 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.696108 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-local-certs\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.696890 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.696425 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.699223 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.699195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772qn\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-kube-api-access-772qn\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.700135 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.700112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-token\") pod \"istiod-openshift-gateway-7574459656-zmckx\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.769974 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.769889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:19.791167 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.791139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mrc\" (UniqueName: \"kubernetes.io/projected/a3210330-e30d-4248-8a9f-5dc79531d1ef-kube-api-access-g5mrc\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:19.791303 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.791190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3210330-e30d-4248-8a9f-5dc79531d1ef-tls-certs\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:19.791373 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:19.791328 2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 19 15:04:19.791431 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:19.791397 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3210330-e30d-4248-8a9f-5dc79531d1ef-tls-certs podName:a3210330-e30d-4248-8a9f-5dc79531d1ef nodeName:}" failed. No retries permitted until 2026-07-19 15:04:20.291372062 +0000 UTC m=+411.155647072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a3210330-e30d-4248-8a9f-5dc79531d1ef-tls-certs") pod "model-serving-api-54d89bd5fc-57bzz" (UID: "a3210330-e30d-4248-8a9f-5dc79531d1ef") : secret "model-serving-api-tls" not found Jul 19 15:04:19.803459 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.803417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mrc\" (UniqueName: \"kubernetes.io/projected/a3210330-e30d-4248-8a9f-5dc79531d1ef-kube-api-access-g5mrc\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:19.901990 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.901908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:19.943694 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:19.943562 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zmckx"] Jul 19 15:04:19.946137 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:19.946109 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aff08c4_9919_4e7f_8add_a44cbe24d7a2.slice/crio-f5a99e0097d02d6205a21bbd8146b28af5560acf01f8a8c8cd8072a685f20801 WatchSource:0}: Error finding container f5a99e0097d02d6205a21bbd8146b28af5560acf01f8a8c8cd8072a685f20801: Status 404 returned error can't find the container with id f5a99e0097d02d6205a21bbd8146b28af5560acf01f8a8c8cd8072a685f20801 Jul 19 15:04:20.297290 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:20.297250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3210330-e30d-4248-8a9f-5dc79531d1ef-tls-certs\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:20.300430 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:20.300396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3210330-e30d-4248-8a9f-5dc79531d1ef-tls-certs\") pod \"model-serving-api-54d89bd5fc-57bzz\" (UID: \"a3210330-e30d-4248-8a9f-5dc79531d1ef\") " pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:20.464740 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:20.464700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:20.506509 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:20.506474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:20.662780 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:20.662739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" event={"ID":"2aff08c4-9919-4e7f-8add-a44cbe24d7a2","Type":"ContainerStarted","Data":"f5a99e0097d02d6205a21bbd8146b28af5560acf01f8a8c8cd8072a685f20801"} Jul 19 15:04:20.678435 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:20.678404 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-57bzz"] Jul 19 15:04:21.260772 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:21.260734 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3210330_e30d_4248_8a9f_5dc79531d1ef.slice/crio-0c917fa855d7472b356ef63c4108c2d33e85bc357ef3ed5aae7d886aba65eb19 WatchSource:0}: Error finding container 0c917fa855d7472b356ef63c4108c2d33e85bc357ef3ed5aae7d886aba65eb19: Status 404 returned error can't find the container with id 0c917fa855d7472b356ef63c4108c2d33e85bc357ef3ed5aae7d886aba65eb19 Jul 19 15:04:21.669681 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:21.669634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" event={"ID":"a3210330-e30d-4248-8a9f-5dc79531d1ef","Type":"ContainerStarted","Data":"0c917fa855d7472b356ef63c4108c2d33e85bc357ef3ed5aae7d886aba65eb19"} Jul 19 15:04:23.681492 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:23.681441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" event={"ID":"76cfdf0d-eef2-4908-bf8f-3403c4aec4b7","Type":"ContainerStarted","Data":"be00128ae3c82984f1a8cd4e07ab1235df6ab6c93793e4aade2bbc3d5b648a56"} Jul 19 15:04:23.683209 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:23.683175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" event={"ID":"0c8f1435-8393-46f7-b8df-7a696a2863d6","Type":"ContainerStarted","Data":"af636acd7e085511b91869a2791274a0ae2700d3d8627ee2c697a362a9e7eb72"} Jul 19 15:04:23.683342 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:23.683272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:23.698324 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:23.698266 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-647d897b75-94xbq" podStartSLOduration=1.95136204 podStartE2EDuration="5.69824958s" totalImagesPullingTime="3.74688754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:23.696860366 +0000 UTC m=+414.561135164" watchObservedRunningTime="2026-07-19 15:04:23.69824958 +0000 UTC m=+414.562524377" Jul 19 15:04:23.713186 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:23.713137 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" podStartSLOduration=2.179816188 podStartE2EDuration="5.713115224s" totalImagesPullingTime="3.533299036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:23.710542763 +0000 UTC m=+414.574817608" watchObservedRunningTime="2026-07-19 15:04:23.713115224 +0000 UTC m=+414.577390020" Jul 19 15:04:24.344414 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.344328 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:04:24.344543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.344466 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:04:24.689860 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.689823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" event={"ID":"2aff08c4-9919-4e7f-8add-a44cbe24d7a2","Type":"ContainerStarted","Data":"b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668"} Jul 19 15:04:24.690306 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.689904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:24.691560 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.691526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" event={"ID":"a3210330-e30d-4248-8a9f-5dc79531d1ef","Type":"ContainerStarted","Data":"63ed28b5f591ed123d07cb592ddbcd1837f6718a40a5209c62c810a6e0fd3faa"} Jul 19 15:04:24.691878 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.691855 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:04:24.710681 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.710632 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" podStartSLOduration=1.315240794 podStartE2EDuration="5.710611163s" totalImagesPullingTime="4.395370369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:24.708938805 +0000 UTC m=+415.573213631" watchObservedRunningTime="2026-07-19 15:04:24.710611163 +0000 UTC m=+415.574885963" Jul 19 15:04:24.735066 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:24.735011 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" podStartSLOduration=2.652407429 podStartE2EDuration="5.734993903s" totalImagesPullingTime="3.082586474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:24.733785027 +0000 UTC m=+415.598059840" watchObservedRunningTime="2026-07-19 15:04:24.734993903 +0000 UTC m=+415.599268699" Jul 19 15:04:25.590758 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:25.590717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:25.698137 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:25.698102 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zmckx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 19 15:04:25.698566 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:25.698160 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" podUID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 15:04:25.703836 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:25.703809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:26.359897 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.359864 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh"] Jul 19 15:04:26.363315 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.363291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.366333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.366304 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kvffn\"" Jul 19 15:04:26.392539 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.392509 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh"] Jul 19 15:04:26.431759 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.431725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:26.559756 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.559713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.559926 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.559770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9r4\" (UniqueName: \"kubernetes.io/projected/0b734a60-94d8-4205-8d3f-f71aef853f9e-kube-api-access-dc9r4\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.559926 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.559851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.559926 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.559881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.559926 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.559909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b734a60-94d8-4205-8d3f-f71aef853f9e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.560076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.559973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.560076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.560023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.560076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.560045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.560076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.560064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.563517 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.563482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:26.661127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9r4\" (UniqueName: \"kubernetes.io/projected/0b734a60-94d8-4205-8d3f-f71aef853f9e-kube-api-access-dc9r4\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b734a60-94d8-4205-8d3f-f71aef853f9e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661783 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.661783 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.661760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.662072 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.662050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b734a60-94d8-4205-8d3f-f71aef853f9e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.663872 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.663846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:26.663989 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.663883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.664073 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.664055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.669307 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.669290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b734a60-94d8-4205-8d3f-f71aef853f9e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.669514 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.669491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9r4\" (UniqueName: \"kubernetes.io/projected/0b734a60-94d8-4205-8d3f-f71aef853f9e-kube-api-access-dc9r4\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4g6hxh\" (UID: \"0b734a60-94d8-4205-8d3f-f71aef853f9e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.673267 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.673247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:26.811615 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:26.811573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh"] Jul 19 15:04:26.813733 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:26.813706 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b734a60_94d8_4205_8d3f_f71aef853f9e.slice/crio-1cb2c526822033b0f9400cb5e20f646c2256dfa21285380245276dfe0aa8fb97 WatchSource:0}: Error finding container 1cb2c526822033b0f9400cb5e20f646c2256dfa21285380245276dfe0aa8fb97: Status 404 returned error can't find the container with id 1cb2c526822033b0f9400cb5e20f646c2256dfa21285380245276dfe0aa8fb97 Jul 19 15:04:27.707377 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:27.707344 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" event={"ID":"0b734a60-94d8-4205-8d3f-f71aef853f9e","Type":"ContainerStarted","Data":"1cb2c526822033b0f9400cb5e20f646c2256dfa21285380245276dfe0aa8fb97"} Jul 19 15:04:28.270724 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.270694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:28.314345 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.314314 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4"] Jul 19 15:04:28.317318 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.317298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.321104 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.321081 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 19 15:04:28.322641 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.322618 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jss4l\"" Jul 19 15:04:28.340114 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.340066 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj"] Jul 19 15:04:28.344547 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.344511 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4"] Jul 19 15:04:28.344706 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.344688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:28.347850 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.347817 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wpkt6\"" Jul 19 15:04:28.348202 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.348181 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 19 15:04:28.368962 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.368871 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj"] Jul 19 15:04:28.374655 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.374626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2627744-13cd-4b05-881c-b3cf99960928-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.374929 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.374905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxf8\" (UniqueName: \"kubernetes.io/projected/e2627744-13cd-4b05-881c-b3cf99960928-kube-api-access-mmxf8\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.385304 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.385267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:28.475372 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.475337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5bz\" (UniqueName: \"kubernetes.io/projected/877b8a5c-8140-4e48-b08b-3c2900ab4815-kube-api-access-qf5bz\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:28.475554 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.475409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxf8\" (UniqueName: \"kubernetes.io/projected/e2627744-13cd-4b05-881c-b3cf99960928-kube-api-access-mmxf8\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.475554 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.475475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877b8a5c-8140-4e48-b08b-3c2900ab4815-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:28.475734 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.475605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2627744-13cd-4b05-881c-b3cf99960928-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.475805 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:28.475734 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 19 15:04:28.475805 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:28.475795 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2627744-13cd-4b05-881c-b3cf99960928-cert podName:e2627744-13cd-4b05-881c-b3cf99960928 nodeName:}" failed. No retries permitted until 2026-07-19 15:04:28.97577293 +0000 UTC m=+419.840047713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e2627744-13cd-4b05-881c-b3cf99960928-cert") pod "kserve-controller-manager-d6f4b65b8-zpqw4" (UID: "e2627744-13cd-4b05-881c-b3cf99960928") : secret "kserve-webhook-server-cert" not found Jul 19 15:04:28.486663 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.486626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxf8\" (UniqueName: \"kubernetes.io/projected/e2627744-13cd-4b05-881c-b3cf99960928-kube-api-access-mmxf8\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.575399 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.575306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:28.576382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.576351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5bz\" (UniqueName: \"kubernetes.io/projected/877b8a5c-8140-4e48-b08b-3c2900ab4815-kube-api-access-qf5bz\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:28.576471 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.576418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877b8a5c-8140-4e48-b08b-3c2900ab4815-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:28.576605 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:28.576562 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 19 15:04:28.576662 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:28.576651 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877b8a5c-8140-4e48-b08b-3c2900ab4815-cert podName:877b8a5c-8140-4e48-b08b-3c2900ab4815 nodeName:}" failed. No retries permitted until 2026-07-19 15:04:29.076635869 +0000 UTC m=+419.940910643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/877b8a5c-8140-4e48-b08b-3c2900ab4815-cert") pod "llmisvc-controller-manager-777c6bd7d5-87vjj" (UID: "877b8a5c-8140-4e48-b08b-3c2900ab4815") : secret "llmisvc-webhook-server-cert" not found Jul 19 15:04:28.594778 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.594749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5bz\" (UniqueName: \"kubernetes.io/projected/877b8a5c-8140-4e48-b08b-3c2900ab4815-kube-api-access-qf5bz\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:28.697848 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.697815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:28.699191 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.699161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:28.980066 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.980008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2627744-13cd-4b05-881c-b3cf99960928-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:28.982838 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:28.982811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2627744-13cd-4b05-881c-b3cf99960928-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpqw4\" (UID: \"e2627744-13cd-4b05-881c-b3cf99960928\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:29.081172 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.081132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877b8a5c-8140-4e48-b08b-3c2900ab4815-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:29.083907 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.083879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877b8a5c-8140-4e48-b08b-3c2900ab4815-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-87vjj\" (UID: \"877b8a5c-8140-4e48-b08b-3c2900ab4815\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:29.229732 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.229689 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:29.258454 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.258418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:29.270638 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.270609 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zmckx"] Jul 19 15:04:29.270801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.270779 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" podUID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" containerName="discovery" containerID="cri-o://b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668" gracePeriod=30 Jul 19 15:04:29.562387 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.562244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:29.662200 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.662170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:29.722560 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.722511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:29.882296 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.882244 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:04:29.882423 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.882352 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:04:29.882423 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.882392 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:04:29.987995 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:29.987966 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:29.998800 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:29.998774 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2627744_13cd_4b05_881c_b3cf99960928.slice/crio-25d10bc886a364629a382417fb5736bfb81f89e0623ee51ac106a3510fde6391 WatchSource:0}: Error finding container 25d10bc886a364629a382417fb5736bfb81f89e0623ee51ac106a3510fde6391: Status 404 returned error can't find the container with id 25d10bc886a364629a382417fb5736bfb81f89e0623ee51ac106a3510fde6391 Jul 19 15:04:30.003450 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.003429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4"] Jul 19 15:04:30.022610 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.022563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj"] Jul 19 15:04:30.024913 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:04:30.024887 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod877b8a5c_8140_4e48_b08b_3c2900ab4815.slice/crio-c0415cdf8ef501290b5a7f7a01a13798bdcaffd7d4f8a7e7c862c4f2c3ef6a49 WatchSource:0}: Error finding container c0415cdf8ef501290b5a7f7a01a13798bdcaffd7d4f8a7e7c862c4f2c3ef6a49: Status 404 returned error can't find the container with id c0415cdf8ef501290b5a7f7a01a13798bdcaffd7d4f8a7e7c862c4f2c3ef6a49 Jul 19 15:04:30.089208 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089107 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-dns-cert\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.089208 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089161 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-kubeconfig\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.089436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089218 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-772qn\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-kube-api-access-772qn\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.089436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089289 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-local-certs\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.089436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089317 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-ca-configmap\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.089436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089379 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-token\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.089436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.089406 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-cacerts\") pod \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\" (UID: \"2aff08c4-9919-4e7f-8add-a44cbe24d7a2\") " Jul 19 15:04:30.090063 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.090005 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:04:30.092178 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.092137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:04:30.092277 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.092252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:04:30.092357 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.092337 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-local-certs" (OuterVolumeSpecName: "local-certs") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:04:30.092401 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.092337 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-kube-api-access-772qn" (OuterVolumeSpecName: "kube-api-access-772qn") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "kube-api-access-772qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:04:30.092498 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.092479 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-cacerts" (OuterVolumeSpecName: "cacerts") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:04:30.092845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.092820 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "2aff08c4-9919-4e7f-8add-a44cbe24d7a2" (UID: "2aff08c4-9919-4e7f-8add-a44cbe24d7a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:04:30.190058 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190018 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-local-certs\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.190058 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190053 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-ca-configmap\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.190281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190077 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-token\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.190281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190089 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-cacerts\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.190281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190101 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-csr-dns-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.190281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190111 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-istio-kubeconfig\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.190281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.190121 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772qn\" (UniqueName: \"kubernetes.io/projected/2aff08c4-9919-4e7f-8add-a44cbe24d7a2-kube-api-access-772qn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:04:30.724921 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.724881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" event={"ID":"e2627744-13cd-4b05-881c-b3cf99960928","Type":"ContainerStarted","Data":"25d10bc886a364629a382417fb5736bfb81f89e0623ee51ac106a3510fde6391"} Jul 19 15:04:30.726330 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.726293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" event={"ID":"877b8a5c-8140-4e48-b08b-3c2900ab4815","Type":"ContainerStarted","Data":"c0415cdf8ef501290b5a7f7a01a13798bdcaffd7d4f8a7e7c862c4f2c3ef6a49"} Jul 19 15:04:30.728114 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.728084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" event={"ID":"0b734a60-94d8-4205-8d3f-f71aef853f9e","Type":"ContainerStarted","Data":"ee11dfeafd333b694cc40f9b5186e5c20fdd7d028c0212985d700bd55518053c"} Jul 19 15:04:30.729435 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.729408 2580 generic.go:356] "Generic (PLEG): container finished" podID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" containerID="b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668" exitCode=0 Jul 19 15:04:30.729553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.729447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" event={"ID":"2aff08c4-9919-4e7f-8add-a44cbe24d7a2","Type":"ContainerDied","Data":"b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668"} Jul 19 15:04:30.729553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.729452 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" Jul 19 15:04:30.729553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.729476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zmckx" event={"ID":"2aff08c4-9919-4e7f-8add-a44cbe24d7a2","Type":"ContainerDied","Data":"f5a99e0097d02d6205a21bbd8146b28af5560acf01f8a8c8cd8072a685f20801"} Jul 19 15:04:30.729553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.729501 2580 scope.go:117] "RemoveContainer" containerID="b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668" Jul 19 15:04:30.740294 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.740273 2580 scope.go:117] "RemoveContainer" containerID="b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668" Jul 19 15:04:30.740898 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:04:30.740865 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668\": container with ID starting with b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668 not found: ID does not exist" containerID="b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668" Jul 19 15:04:30.741005 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.740911 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668"} err="failed to get container status \"b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668\": rpc error: code = NotFound desc = could not find container \"b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668\": container with ID starting with b61d533b39c9fb3bb7bb34c26440c1614a6dfd56dccf65f937c4d35f4351d668 not found: ID does not exist" Jul 19 15:04:30.751854 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.751801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" podStartSLOduration=1.685925264 podStartE2EDuration="4.751782405s" totalImagesPullingTime="3.065857141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:30.748804082 +0000 UTC m=+421.613078879" watchObservedRunningTime="2026-07-19 15:04:30.751782405 +0000 UTC m=+421.616057202" Jul 19 15:04:30.762735 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.762706 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zmckx"] Jul 19 15:04:30.766217 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:30.766189 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zmckx"] Jul 19 15:04:31.668844 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:31.668805 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" path="/var/lib/kubelet/pods/2aff08c4-9919-4e7f-8add-a44cbe24d7a2/volumes" Jul 19 15:04:31.673447 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:31.673421 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:31.679192 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:31.679171 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:31.736724 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:31.736692 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:31.737641 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:31.737619 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4g6hxh" Jul 19 15:04:32.743917 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:32.743870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" event={"ID":"e2627744-13cd-4b05-881c-b3cf99960928","Type":"ContainerStarted","Data":"d1deab2f45671be76a2f60f777922010b1b96e371d7c498c72ff9562ab17a45c"} Jul 19 15:04:32.744374 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:32.744091 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:04:32.763160 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:32.763109 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" podStartSLOduration=2.220552594 podStartE2EDuration="4.763093396s" totalImagesPullingTime="2.542540802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:32.761395453 +0000 UTC m=+423.625670250" watchObservedRunningTime="2026-07-19 15:04:32.763093396 +0000 UTC m=+423.627368194" Jul 19 15:04:33.295733 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:33.295695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:04:33.750730 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:33.750694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" event={"ID":"877b8a5c-8140-4e48-b08b-3c2900ab4815","Type":"ContainerStarted","Data":"4935a1dfe75cc27b6f7e34b6eb1f34331f82a291752d32c6ce563a9c4fff7ff8"} Jul 19 15:04:33.751186 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:33.751039 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:04:33.768129 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:33.768077 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" podStartSLOduration=2.384806772 podStartE2EDuration="5.768061086s" totalImagesPullingTime="3.383254314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:33.766272571 +0000 UTC m=+424.630547372" watchObservedRunningTime="2026-07-19 15:04:33.768061086 +0000 UTC m=+424.632335883" Jul 19 15:04:34.694166 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:34.694136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rp424" Jul 19 15:04:36.705739 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:04:36.705709 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-57bzz" Jul 19 15:05:03.756774 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:03.756695 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpqw4" Jul 19 15:05:04.757834 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:04.757804 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-87vjj" Jul 19 15:05:19.207993 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.207954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:19.306435 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.306390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:19.347316 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.347274 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dqwkn"] Jul 19 15:05:19.347810 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.347781 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" containerName="discovery" Jul 19 15:05:19.347952 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.347894 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" containerName="discovery" Jul 19 15:05:19.348037 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.348013 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aff08c4-9919-4e7f-8add-a44cbe24d7a2" containerName="discovery" Jul 19 15:05:19.368968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.367221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dqwkn"] Jul 19 15:05:19.368968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.367366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:19.370374 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.370328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 15:05:19.371102 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.371080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 15:05:19.371270 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.371106 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7g8vk\"" Jul 19 15:05:19.389431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.389403 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:19.510257 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.510164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8clk\" (UniqueName: \"kubernetes.io/projected/c195ec4f-30e2-4d2b-b63b-33446190a56e-kube-api-access-v8clk\") pod \"kuadrant-operator-catalog-dqwkn\" (UID: \"c195ec4f-30e2-4d2b-b63b-33446190a56e\") " pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:19.611128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.611090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8clk\" (UniqueName: \"kubernetes.io/projected/c195ec4f-30e2-4d2b-b63b-33446190a56e-kube-api-access-v8clk\") pod \"kuadrant-operator-catalog-dqwkn\" (UID: \"c195ec4f-30e2-4d2b-b63b-33446190a56e\") " pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:19.619220 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.619195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8clk\" (UniqueName: \"kubernetes.io/projected/c195ec4f-30e2-4d2b-b63b-33446190a56e-kube-api-access-v8clk\") pod \"kuadrant-operator-catalog-dqwkn\" (UID: \"c195ec4f-30e2-4d2b-b63b-33446190a56e\") " pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:19.678017 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.677987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:19.716012 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.715973 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dqwkn"] Jul 19 15:05:19.837398 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.837372 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dqwkn"] Jul 19 15:05:19.839761 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:19.839729 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc195ec4f_30e2_4d2b_b63b_33446190a56e.slice/crio-a6e2cdb2612b159b2bacf09d4c9fc98e012fe376dfc89f5e949558fe8e30e263 WatchSource:0}: Error finding container a6e2cdb2612b159b2bacf09d4c9fc98e012fe376dfc89f5e949558fe8e30e263: Status 404 returned error can't find the container with id a6e2cdb2612b159b2bacf09d4c9fc98e012fe376dfc89f5e949558fe8e30e263 Jul 19 15:05:19.866126 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.866090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:19.922338 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.922299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:19.928633 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.928597 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b94l7"] Jul 19 15:05:19.933310 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.933289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:19.938362 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.938334 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b94l7"] Jul 19 15:05:19.975436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:19.975404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" event={"ID":"c195ec4f-30e2-4d2b-b63b-33446190a56e","Type":"ContainerStarted","Data":"a6e2cdb2612b159b2bacf09d4c9fc98e012fe376dfc89f5e949558fe8e30e263"} Jul 19 15:05:20.115949 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:20.115922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrsj2\" (UniqueName: \"kubernetes.io/projected/2350647f-1aae-40b7-885d-fac03cdee35c-kube-api-access-hrsj2\") pod \"kuadrant-operator-catalog-b94l7\" (UID: \"2350647f-1aae-40b7-885d-fac03cdee35c\") " pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:20.216832 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:20.216793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrsj2\" (UniqueName: \"kubernetes.io/projected/2350647f-1aae-40b7-885d-fac03cdee35c-kube-api-access-hrsj2\") pod \"kuadrant-operator-catalog-b94l7\" (UID: \"2350647f-1aae-40b7-885d-fac03cdee35c\") " pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:20.224525 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:20.224490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrsj2\" (UniqueName: \"kubernetes.io/projected/2350647f-1aae-40b7-885d-fac03cdee35c-kube-api-access-hrsj2\") pod \"kuadrant-operator-catalog-b94l7\" (UID: \"2350647f-1aae-40b7-885d-fac03cdee35c\") " pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:20.245500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:20.245464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:20.374444 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:20.374420 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-b94l7"] Jul 19 15:05:20.376344 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:20.376319 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2350647f_1aae_40b7_885d_fac03cdee35c.slice/crio-0f5b8851fc4c7aca8a666b439049df8374306467a65304772b08fb212421edb7 WatchSource:0}: Error finding container 0f5b8851fc4c7aca8a666b439049df8374306467a65304772b08fb212421edb7: Status 404 returned error can't find the container with id 0f5b8851fc4c7aca8a666b439049df8374306467a65304772b08fb212421edb7 Jul 19 15:05:20.984330 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:20.984256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" event={"ID":"2350647f-1aae-40b7-885d-fac03cdee35c","Type":"ContainerStarted","Data":"0f5b8851fc4c7aca8a666b439049df8374306467a65304772b08fb212421edb7"} Jul 19 15:05:21.822415 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:21.822382 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:21.991785 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:21.991669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" event={"ID":"c195ec4f-30e2-4d2b-b63b-33446190a56e","Type":"ContainerStarted","Data":"ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d"} Jul 19 15:05:21.991980 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:21.991867 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" podUID="c195ec4f-30e2-4d2b-b63b-33446190a56e" containerName="registry-server" containerID="cri-o://ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d" gracePeriod=2 Jul 19 15:05:21.993696 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:21.993669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" event={"ID":"2350647f-1aae-40b7-885d-fac03cdee35c","Type":"ContainerStarted","Data":"bd8dc144eb4529386454a3021681f035bd294487913db8de56d6875b9fb6a81e"} Jul 19 15:05:22.006228 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:22.006173 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" podStartSLOduration=1.004388813 podStartE2EDuration="3.006151606s" totalImagesPullingTime="2.001762793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:05:22.005128266 +0000 UTC m=+472.869403066" watchObservedRunningTime="2026-07-19 15:05:22.006151606 +0000 UTC m=+472.870426383" Jul 19 15:05:22.020386 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:22.020335 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" podStartSLOduration=1.554301137 podStartE2EDuration="3.020317994s" totalImagesPullingTime="1.466016857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:05:22.018704369 +0000 UTC m=+472.882979169" watchObservedRunningTime="2026-07-19 15:05:22.020317994 +0000 UTC m=+472.884592823" Jul 19 15:05:22.237635 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:22.237610 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:22.337662 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:22.337629 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8clk\" (UniqueName: \"kubernetes.io/projected/c195ec4f-30e2-4d2b-b63b-33446190a56e-kube-api-access-v8clk\") pod \"c195ec4f-30e2-4d2b-b63b-33446190a56e\" (UID: \"c195ec4f-30e2-4d2b-b63b-33446190a56e\") " Jul 19 15:05:22.339843 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:22.339814 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c195ec4f-30e2-4d2b-b63b-33446190a56e-kube-api-access-v8clk" (OuterVolumeSpecName: "kube-api-access-v8clk") pod "c195ec4f-30e2-4d2b-b63b-33446190a56e" (UID: "c195ec4f-30e2-4d2b-b63b-33446190a56e"). InnerVolumeSpecName "kube-api-access-v8clk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:05:22.438500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:22.438411 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8clk\" (UniqueName: \"kubernetes.io/projected/c195ec4f-30e2-4d2b-b63b-33446190a56e-kube-api-access-v8clk\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:23.000495 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.000454 2580 generic.go:356] "Generic (PLEG): container finished" podID="c195ec4f-30e2-4d2b-b63b-33446190a56e" containerID="ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d" exitCode=0 Jul 19 15:05:23.001036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.000517 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" Jul 19 15:05:23.001036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.000536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" event={"ID":"c195ec4f-30e2-4d2b-b63b-33446190a56e","Type":"ContainerDied","Data":"ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d"} Jul 19 15:05:23.001036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.000656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dqwkn" event={"ID":"c195ec4f-30e2-4d2b-b63b-33446190a56e","Type":"ContainerDied","Data":"a6e2cdb2612b159b2bacf09d4c9fc98e012fe376dfc89f5e949558fe8e30e263"} Jul 19 15:05:23.001036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.000687 2580 scope.go:117] "RemoveContainer" containerID="ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d" Jul 19 15:05:23.010080 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.010063 2580 scope.go:117] "RemoveContainer" containerID="ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d" Jul 19 15:05:23.010331 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:05:23.010315 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d\": container with ID starting with ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d not found: ID does not exist" containerID="ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d" Jul 19 15:05:23.010382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.010340 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d"} err="failed to get container status \"ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d\": rpc error: code = NotFound desc = could not find container \"ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d\": container with ID starting with ad276916f01797353d10d0b79a0b9e403891c94c4b930c25d3febc0e9d61045d not found: ID does not exist" Jul 19 15:05:23.020856 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.020832 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dqwkn"] Jul 19 15:05:23.024499 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.024472 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dqwkn"] Jul 19 15:05:23.662304 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:23.662269 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c195ec4f-30e2-4d2b-b63b-33446190a56e" path="/var/lib/kubelet/pods/c195ec4f-30e2-4d2b-b63b-33446190a56e/volumes" Jul 19 15:05:27.652992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:27.652954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:27.713280 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:27.713243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:30.245951 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:30.245914 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:30.246333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:30.245969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:30.268068 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:30.268045 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:31.062307 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:31.062281 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-b94l7" Jul 19 15:05:34.730186 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.730157 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q"] Jul 19 15:05:34.730553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.730486 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c195ec4f-30e2-4d2b-b63b-33446190a56e" containerName="registry-server" Jul 19 15:05:34.730553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.730497 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c195ec4f-30e2-4d2b-b63b-33446190a56e" containerName="registry-server" Jul 19 15:05:34.730553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.730553 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c195ec4f-30e2-4d2b-b63b-33446190a56e" containerName="registry-server" Jul 19 15:05:34.734804 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.734788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.736736 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.736714 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7rbnp\"" Jul 19 15:05:34.740431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.740413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q"] Jul 19 15:05:34.823538 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.823502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.823736 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.823551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.823736 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.823611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhn9\" (UniqueName: \"kubernetes.io/projected/249af867-feba-439f-b860-da70ac3387d2-kube-api-access-bfhn9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.924185 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.924147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.924185 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.924188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.924380 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.924208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhn9\" (UniqueName: \"kubernetes.io/projected/249af867-feba-439f-b860-da70ac3387d2-kube-api-access-bfhn9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.924602 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.924564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.924701 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.924661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:34.944837 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:34.944808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhn9\" (UniqueName: \"kubernetes.io/projected/249af867-feba-439f-b860-da70ac3387d2-kube-api-access-bfhn9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:35.044818 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.044729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:35.177598 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.177558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q"] Jul 19 15:05:35.179283 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:35.179256 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249af867_feba_439f_b860_da70ac3387d2.slice/crio-a22ff4ee84b6216adacfecf6201b325fb91070cf3863586b8a2f099a092bd95a WatchSource:0}: Error finding container a22ff4ee84b6216adacfecf6201b325fb91070cf3863586b8a2f099a092bd95a: Status 404 returned error can't find the container with id a22ff4ee84b6216adacfecf6201b325fb91070cf3863586b8a2f099a092bd95a Jul 19 15:05:35.532019 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.531986 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992"] Jul 19 15:05:35.535505 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.535488 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.543442 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.543414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992"] Jul 19 15:05:35.630258 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.630220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.630433 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.630280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.630433 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.630302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhjt\" (UniqueName: \"kubernetes.io/projected/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-kube-api-access-2nhjt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.731206 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.731170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.731206 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.731204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhjt\" (UniqueName: \"kubernetes.io/projected/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-kube-api-access-2nhjt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.731784 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.731331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.731784 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.731656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.731784 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.731656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.739357 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.739332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhjt\" (UniqueName: \"kubernetes.io/projected/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-kube-api-access-2nhjt\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.845325 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.845249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:35.978741 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:35.978717 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992"] Jul 19 15:05:35.980676 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:35.980642 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fd2a43_9ad8_4b8a_b023_687f4bfff9cd.slice/crio-57b1708a80a8243a82717da54b2d30f5de91bc0ef967ffb2a958f8b0dc38a8f3 WatchSource:0}: Error finding container 57b1708a80a8243a82717da54b2d30f5de91bc0ef967ffb2a958f8b0dc38a8f3: Status 404 returned error can't find the container with id 57b1708a80a8243a82717da54b2d30f5de91bc0ef967ffb2a958f8b0dc38a8f3 Jul 19 15:05:36.065960 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.065915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" event={"ID":"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd","Type":"ContainerStarted","Data":"c2381b0b4c7f5487397b08bb3f25c0da27941c7a3241e75201b19a7a6c0f19b7"} Jul 19 15:05:36.066152 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.065984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" event={"ID":"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd","Type":"ContainerStarted","Data":"57b1708a80a8243a82717da54b2d30f5de91bc0ef967ffb2a958f8b0dc38a8f3"} Jul 19 15:05:36.067547 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.067522 2580 generic.go:356] "Generic (PLEG): container finished" podID="249af867-feba-439f-b860-da70ac3387d2" containerID="2e5f73f35acb810001acf09ac0d01389977f3693e15a4707b27b0eaf8b869a0c" exitCode=0 Jul 19 15:05:36.067676 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.067606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" event={"ID":"249af867-feba-439f-b860-da70ac3387d2","Type":"ContainerDied","Data":"2e5f73f35acb810001acf09ac0d01389977f3693e15a4707b27b0eaf8b869a0c"} Jul 19 15:05:36.067676 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.067656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" event={"ID":"249af867-feba-439f-b860-da70ac3387d2","Type":"ContainerStarted","Data":"a22ff4ee84b6216adacfecf6201b325fb91070cf3863586b8a2f099a092bd95a"} Jul 19 15:05:36.133750 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.133715 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb"] Jul 19 15:05:36.137698 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.137677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.144250 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.144226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb"] Jul 19 15:05:36.234895 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.234864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.234895 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.234899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.235094 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.234965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqv6\" (UniqueName: \"kubernetes.io/projected/b5cea9a8-a913-4a95-bf98-6c42dba66e35-kube-api-access-5mqv6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.335656 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.335619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqv6\" (UniqueName: \"kubernetes.io/projected/b5cea9a8-a913-4a95-bf98-6c42dba66e35-kube-api-access-5mqv6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.335845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.335680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.335845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.335724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.336003 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.335986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.336096 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.336078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.343332 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.343300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqv6\" (UniqueName: \"kubernetes.io/projected/b5cea9a8-a913-4a95-bf98-6c42dba66e35-kube-api-access-5mqv6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.464194 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.464152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:36.536805 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.536773 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t"] Jul 19 15:05:36.541844 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.541820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.547365 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.547106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t"] Jul 19 15:05:36.601003 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:36.600957 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cea9a8_a913_4a95_bf98_6c42dba66e35.slice/crio-ea954e99be086e6d58573000e91b60e6a317c8d98f23e929861e69f6579b4193 WatchSource:0}: Error finding container ea954e99be086e6d58573000e91b60e6a317c8d98f23e929861e69f6579b4193: Status 404 returned error can't find the container with id ea954e99be086e6d58573000e91b60e6a317c8d98f23e929861e69f6579b4193 Jul 19 15:05:36.604166 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.604141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb"] Jul 19 15:05:36.637688 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.637662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.637813 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.637781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hh6\" (UniqueName: \"kubernetes.io/projected/338e367b-0cee-411c-9f74-e26eef0dc377-kube-api-access-x8hh6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.637917 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.637835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.738244 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.738164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.738244 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.738234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hh6\" (UniqueName: \"kubernetes.io/projected/338e367b-0cee-411c-9f74-e26eef0dc377-kube-api-access-x8hh6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.738634 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.738263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.738634 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.738520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.738634 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.738552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.745712 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.745686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hh6\" (UniqueName: \"kubernetes.io/projected/338e367b-0cee-411c-9f74-e26eef0dc377-kube-api-access-x8hh6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.854688 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.854649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:36.989781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:36.989750 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t"] Jul 19 15:05:36.991434 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:36.991408 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338e367b_0cee_411c_9f74_e26eef0dc377.slice/crio-1e81748d0558eae60f8337af309af9b2d9f79ef1722c7fd68e4b959e55b9744a WatchSource:0}: Error finding container 1e81748d0558eae60f8337af309af9b2d9f79ef1722c7fd68e4b959e55b9744a: Status 404 returned error can't find the container with id 1e81748d0558eae60f8337af309af9b2d9f79ef1722c7fd68e4b959e55b9744a Jul 19 15:05:37.074330 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.074295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerStarted","Data":"bef68890b334950bcdb736cff5c43c7d35be9a8b36b2dfb63347c2b928489507"} Jul 19 15:05:37.074495 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.074352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerStarted","Data":"1e81748d0558eae60f8337af309af9b2d9f79ef1722c7fd68e4b959e55b9744a"} Jul 19 15:05:37.075955 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.075932 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerID="4d7e72d1906a2a3db7c0a6a1405ddbffe3e9e4654e91381e85fdd4039b7a9a25" exitCode=0 Jul 19 15:05:37.076086 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.075993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" event={"ID":"b5cea9a8-a913-4a95-bf98-6c42dba66e35","Type":"ContainerDied","Data":"4d7e72d1906a2a3db7c0a6a1405ddbffe3e9e4654e91381e85fdd4039b7a9a25"} Jul 19 15:05:37.076086 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.076035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" event={"ID":"b5cea9a8-a913-4a95-bf98-6c42dba66e35","Type":"ContainerStarted","Data":"ea954e99be086e6d58573000e91b60e6a317c8d98f23e929861e69f6579b4193"} Jul 19 15:05:37.077836 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.077815 2580 generic.go:356] "Generic (PLEG): container finished" podID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerID="c2381b0b4c7f5487397b08bb3f25c0da27941c7a3241e75201b19a7a6c0f19b7" exitCode=0 Jul 19 15:05:37.077925 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.077855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" event={"ID":"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd","Type":"ContainerDied","Data":"c2381b0b4c7f5487397b08bb3f25c0da27941c7a3241e75201b19a7a6c0f19b7"} Jul 19 15:05:37.083867 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.083846 2580 generic.go:356] "Generic (PLEG): container finished" podID="249af867-feba-439f-b860-da70ac3387d2" containerID="694076ac5cc5877e87cd7fe2e50e8fcfda6d88cf0f96b501dbb86a2934a99307" exitCode=0 Jul 19 15:05:37.083980 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:37.083899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" event={"ID":"249af867-feba-439f-b860-da70ac3387d2","Type":"ContainerDied","Data":"694076ac5cc5877e87cd7fe2e50e8fcfda6d88cf0f96b501dbb86a2934a99307"} Jul 19 15:05:38.091177 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.091138 2580 generic.go:356] "Generic (PLEG): container finished" podID="338e367b-0cee-411c-9f74-e26eef0dc377" containerID="bef68890b334950bcdb736cff5c43c7d35be9a8b36b2dfb63347c2b928489507" exitCode=0 Jul 19 15:05:38.091655 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.091273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerDied","Data":"bef68890b334950bcdb736cff5c43c7d35be9a8b36b2dfb63347c2b928489507"} Jul 19 15:05:38.093648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.093621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" event={"ID":"b5cea9a8-a913-4a95-bf98-6c42dba66e35","Type":"ContainerStarted","Data":"9875c421446c7eeb7aa7e5440ba4ad1e085a529195fc38ee130cddd786cdcfe6"} Jul 19 15:05:38.095946 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.095852 2580 generic.go:356] "Generic (PLEG): container finished" podID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerID="abed01fcb2b0aff1343a7ce6823ae45a9f7efad408eb353377f820a986113adf" exitCode=0 Jul 19 15:05:38.096104 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.095938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" event={"ID":"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd","Type":"ContainerDied","Data":"abed01fcb2b0aff1343a7ce6823ae45a9f7efad408eb353377f820a986113adf"} Jul 19 15:05:38.098642 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.098614 2580 generic.go:356] "Generic (PLEG): container finished" podID="249af867-feba-439f-b860-da70ac3387d2" containerID="aca6052dd2fcbb170c740fdbfb7710698381dae216a9ec746fa152d1326b4afe" exitCode=0 Jul 19 15:05:38.098750 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:38.098668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" event={"ID":"249af867-feba-439f-b860-da70ac3387d2","Type":"ContainerDied","Data":"aca6052dd2fcbb170c740fdbfb7710698381dae216a9ec746fa152d1326b4afe"} Jul 19 15:05:39.106822 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.106795 2580 generic.go:356] "Generic (PLEG): container finished" podID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerID="d0b46e0c5ef886e754a7de21122e7574a472ba6d9747dbbfbc088d9fe335a5ea" exitCode=0 Jul 19 15:05:39.107158 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.106883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" event={"ID":"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd","Type":"ContainerDied","Data":"d0b46e0c5ef886e754a7de21122e7574a472ba6d9747dbbfbc088d9fe335a5ea"} Jul 19 15:05:39.108627 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.108608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerStarted","Data":"044251d1cf09fdc04d50fbfb11d5df0463fc260e7b1844224013913579d15aa0"} Jul 19 15:05:39.110532 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.110509 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerID="9875c421446c7eeb7aa7e5440ba4ad1e085a529195fc38ee130cddd786cdcfe6" exitCode=0 Jul 19 15:05:39.110653 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.110612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" event={"ID":"b5cea9a8-a913-4a95-bf98-6c42dba66e35","Type":"ContainerDied","Data":"9875c421446c7eeb7aa7e5440ba4ad1e085a529195fc38ee130cddd786cdcfe6"} Jul 19 15:05:39.257647 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.257623 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:39.356062 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.356036 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-util\") pod \"249af867-feba-439f-b860-da70ac3387d2\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " Jul 19 15:05:39.356216 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.356071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-bundle\") pod \"249af867-feba-439f-b860-da70ac3387d2\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " Jul 19 15:05:39.356216 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.356109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfhn9\" (UniqueName: \"kubernetes.io/projected/249af867-feba-439f-b860-da70ac3387d2-kube-api-access-bfhn9\") pod \"249af867-feba-439f-b860-da70ac3387d2\" (UID: \"249af867-feba-439f-b860-da70ac3387d2\") " Jul 19 15:05:39.356531 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.356506 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-bundle" (OuterVolumeSpecName: "bundle") pod "249af867-feba-439f-b860-da70ac3387d2" (UID: "249af867-feba-439f-b860-da70ac3387d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:39.358116 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.358095 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249af867-feba-439f-b860-da70ac3387d2-kube-api-access-bfhn9" (OuterVolumeSpecName: "kube-api-access-bfhn9") pod "249af867-feba-439f-b860-da70ac3387d2" (UID: "249af867-feba-439f-b860-da70ac3387d2"). InnerVolumeSpecName "kube-api-access-bfhn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:05:39.360859 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.360836 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-util" (OuterVolumeSpecName: "util") pod "249af867-feba-439f-b860-da70ac3387d2" (UID: "249af867-feba-439f-b860-da70ac3387d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:39.457731 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.457647 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:39.457731 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.457676 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/249af867-feba-439f-b860-da70ac3387d2-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:39.457731 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:39.457686 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhn9\" (UniqueName: \"kubernetes.io/projected/249af867-feba-439f-b860-da70ac3387d2-kube-api-access-bfhn9\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:40.117495 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.117459 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" Jul 19 15:05:40.117909 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.117464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q" event={"ID":"249af867-feba-439f-b860-da70ac3387d2","Type":"ContainerDied","Data":"a22ff4ee84b6216adacfecf6201b325fb91070cf3863586b8a2f099a092bd95a"} Jul 19 15:05:40.117909 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.117619 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22ff4ee84b6216adacfecf6201b325fb91070cf3863586b8a2f099a092bd95a" Jul 19 15:05:40.119330 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.119305 2580 generic.go:356] "Generic (PLEG): container finished" podID="338e367b-0cee-411c-9f74-e26eef0dc377" containerID="044251d1cf09fdc04d50fbfb11d5df0463fc260e7b1844224013913579d15aa0" exitCode=0 Jul 19 15:05:40.119459 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.119390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerDied","Data":"044251d1cf09fdc04d50fbfb11d5df0463fc260e7b1844224013913579d15aa0"} Jul 19 15:05:40.121793 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.121770 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerID="9e9827e9d5d7ef06d5fbbb6cc05e319516aa00901b2e1559872b8199dea0f3e1" exitCode=0 Jul 19 15:05:40.121896 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.121850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" event={"ID":"b5cea9a8-a913-4a95-bf98-6c42dba66e35","Type":"ContainerDied","Data":"9e9827e9d5d7ef06d5fbbb6cc05e319516aa00901b2e1559872b8199dea0f3e1"} Jul 19 15:05:40.261817 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.261789 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:40.366156 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.366120 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-util\") pod \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " Jul 19 15:05:40.366296 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.366180 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-bundle\") pod \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " Jul 19 15:05:40.366296 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.366200 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nhjt\" (UniqueName: \"kubernetes.io/projected/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-kube-api-access-2nhjt\") pod \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\" (UID: \"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd\") " Jul 19 15:05:40.366938 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.366896 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-bundle" (OuterVolumeSpecName: "bundle") pod "63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" (UID: "63fd2a43-9ad8-4b8a-b023-687f4bfff9cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:40.368410 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.368386 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-kube-api-access-2nhjt" (OuterVolumeSpecName: "kube-api-access-2nhjt") pod "63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" (UID: "63fd2a43-9ad8-4b8a-b023-687f4bfff9cd"). InnerVolumeSpecName "kube-api-access-2nhjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:05:40.372122 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.372100 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-util" (OuterVolumeSpecName: "util") pod "63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" (UID: "63fd2a43-9ad8-4b8a-b023-687f4bfff9cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:40.455347 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.455266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:40.467715 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.467689 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:40.467715 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.467710 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:40.467715 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.467720 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nhjt\" (UniqueName: \"kubernetes.io/projected/63fd2a43-9ad8-4b8a-b023-687f4bfff9cd-kube-api-access-2nhjt\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:40.551512 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.551470 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:40.601019 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:40.600982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:41.129176 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.129145 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" Jul 19 15:05:41.129176 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.129168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992" event={"ID":"63fd2a43-9ad8-4b8a-b023-687f4bfff9cd","Type":"ContainerDied","Data":"57b1708a80a8243a82717da54b2d30f5de91bc0ef967ffb2a958f8b0dc38a8f3"} Jul 19 15:05:41.129628 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.129215 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b1708a80a8243a82717da54b2d30f5de91bc0ef967ffb2a958f8b0dc38a8f3" Jul 19 15:05:41.131357 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.131299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:41.131661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.131635 2580 generic.go:356] "Generic (PLEG): container finished" podID="338e367b-0cee-411c-9f74-e26eef0dc377" containerID="e9409ffd4f819b1ebb2b85e2d6049ee43ff690a88a29085609307433e19e9b49" exitCode=0 Jul 19 15:05:41.131768 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.131674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerDied","Data":"e9409ffd4f819b1ebb2b85e2d6049ee43ff690a88a29085609307433e19e9b49"} Jul 19 15:05:41.232425 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.232329 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:41.268064 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.268041 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:41.373297 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.373263 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mqv6\" (UniqueName: \"kubernetes.io/projected/b5cea9a8-a913-4a95-bf98-6c42dba66e35-kube-api-access-5mqv6\") pod \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " Jul 19 15:05:41.373490 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.373328 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-util\") pod \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " Jul 19 15:05:41.373490 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.373395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-bundle\") pod \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\" (UID: \"b5cea9a8-a913-4a95-bf98-6c42dba66e35\") " Jul 19 15:05:41.373904 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.373874 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-bundle" (OuterVolumeSpecName: "bundle") pod "b5cea9a8-a913-4a95-bf98-6c42dba66e35" (UID: "b5cea9a8-a913-4a95-bf98-6c42dba66e35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:41.375509 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.375477 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cea9a8-a913-4a95-bf98-6c42dba66e35-kube-api-access-5mqv6" (OuterVolumeSpecName: "kube-api-access-5mqv6") pod "b5cea9a8-a913-4a95-bf98-6c42dba66e35" (UID: "b5cea9a8-a913-4a95-bf98-6c42dba66e35"). InnerVolumeSpecName "kube-api-access-5mqv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:05:41.378771 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.378752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-util" (OuterVolumeSpecName: "util") pod "b5cea9a8-a913-4a95-bf98-6c42dba66e35" (UID: "b5cea9a8-a913-4a95-bf98-6c42dba66e35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:41.474845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.474810 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mqv6\" (UniqueName: \"kubernetes.io/projected/b5cea9a8-a913-4a95-bf98-6c42dba66e35-kube-api-access-5mqv6\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:41.474845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.474840 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:41.474845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:41.474849 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cea9a8-a913-4a95-bf98-6c42dba66e35-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:42.138489 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.138462 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" Jul 19 15:05:42.138489 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.138479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb" event={"ID":"b5cea9a8-a913-4a95-bf98-6c42dba66e35","Type":"ContainerDied","Data":"ea954e99be086e6d58573000e91b60e6a317c8d98f23e929861e69f6579b4193"} Jul 19 15:05:42.138950 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.138523 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea954e99be086e6d58573000e91b60e6a317c8d98f23e929861e69f6579b4193" Jul 19 15:05:42.274204 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.274181 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:42.381844 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.381813 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8hh6\" (UniqueName: \"kubernetes.io/projected/338e367b-0cee-411c-9f74-e26eef0dc377-kube-api-access-x8hh6\") pod \"338e367b-0cee-411c-9f74-e26eef0dc377\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " Jul 19 15:05:42.382036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.381882 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-util\") pod \"338e367b-0cee-411c-9f74-e26eef0dc377\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " Jul 19 15:05:42.382036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.382000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-bundle\") pod \"338e367b-0cee-411c-9f74-e26eef0dc377\" (UID: \"338e367b-0cee-411c-9f74-e26eef0dc377\") " Jul 19 15:05:42.382416 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.382384 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-bundle" (OuterVolumeSpecName: "bundle") pod "338e367b-0cee-411c-9f74-e26eef0dc377" (UID: "338e367b-0cee-411c-9f74-e26eef0dc377"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:42.384015 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.383996 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338e367b-0cee-411c-9f74-e26eef0dc377-kube-api-access-x8hh6" (OuterVolumeSpecName: "kube-api-access-x8hh6") pod "338e367b-0cee-411c-9f74-e26eef0dc377" (UID: "338e367b-0cee-411c-9f74-e26eef0dc377"). InnerVolumeSpecName "kube-api-access-x8hh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:05:42.476249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.476201 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-util" (OuterVolumeSpecName: "util") pod "338e367b-0cee-411c-9f74-e26eef0dc377" (UID: "338e367b-0cee-411c-9f74-e26eef0dc377"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:05:42.487474 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.487442 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-bundle\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:42.487474 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.487475 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hh6\" (UniqueName: \"kubernetes.io/projected/338e367b-0cee-411c-9f74-e26eef0dc377-kube-api-access-x8hh6\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:42.487924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:42.487486 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/338e367b-0cee-411c-9f74-e26eef0dc377-util\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:05:43.145256 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:43.145228 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" Jul 19 15:05:43.145256 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:43.145250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t" event={"ID":"338e367b-0cee-411c-9f74-e26eef0dc377","Type":"ContainerDied","Data":"1e81748d0558eae60f8337af309af9b2d9f79ef1722c7fd68e4b959e55b9744a"} Jul 19 15:05:43.145818 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:43.145293 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e81748d0558eae60f8337af309af9b2d9f79ef1722c7fd68e4b959e55b9744a" Jul 19 15:05:47.880884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:47.880853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:48.481647 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:48.481612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:51.027717 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:51.027681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:51.128603 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:51.128550 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:52.583395 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583360 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-55mt5"] Jul 19 15:05:52.583781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583737 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="util" Jul 19 15:05:52.583781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583750 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="util" Jul 19 15:05:52.583781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583766 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="pull" Jul 19 15:05:52.583781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583771 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="pull" Jul 19 15:05:52.583781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583777 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="extract" Jul 19 15:05:52.583781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583783 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583792 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="util" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583797 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="util" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583803 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583808 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583815 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="pull" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583822 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="pull" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583828 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="util" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583832 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="util" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583837 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="pull" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583843 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="pull" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583850 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="util" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583854 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="util" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583860 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583866 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583885 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="pull" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583891 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="pull" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583897 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583903 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583955 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="249af867-feba-439f-b860-da70ac3387d2" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583963 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="338e367b-0cee-411c-9f74-e26eef0dc377" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583969 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cea9a8-a913-4a95-bf98-6c42dba66e35" containerName="extract" Jul 19 15:05:52.583971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.583977 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63fd2a43-9ad8-4b8a-b023-687f4bfff9cd" containerName="extract" Jul 19 15:05:52.588552 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.588535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:05:52.591822 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.591802 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hs5n2\"" Jul 19 15:05:52.607995 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.607966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-55mt5"] Jul 19 15:05:52.676031 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.675996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwch6\" (UniqueName: \"kubernetes.io/projected/8ea3c2dd-c626-48ee-aa71-a35912a689d5-kube-api-access-pwch6\") pod \"authorino-operator-75cc8d5cb7-55mt5\" (UID: \"8ea3c2dd-c626-48ee-aa71-a35912a689d5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:05:52.777133 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.777096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwch6\" (UniqueName: \"kubernetes.io/projected/8ea3c2dd-c626-48ee-aa71-a35912a689d5-kube-api-access-pwch6\") pod \"authorino-operator-75cc8d5cb7-55mt5\" (UID: \"8ea3c2dd-c626-48ee-aa71-a35912a689d5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:05:52.789778 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.789744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwch6\" (UniqueName: \"kubernetes.io/projected/8ea3c2dd-c626-48ee-aa71-a35912a689d5-kube-api-access-pwch6\") pod \"authorino-operator-75cc8d5cb7-55mt5\" (UID: \"8ea3c2dd-c626-48ee-aa71-a35912a689d5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:05:52.898907 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:52.898873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:05:53.030165 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:53.030136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-55mt5"] Jul 19 15:05:53.031706 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:53.031669 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea3c2dd_c626_48ee_aa71_a35912a689d5.slice/crio-5609172ca13a2e5356bc4bf5635c276cc41b46f9f32dfc0f8ddb632006897462 WatchSource:0}: Error finding container 5609172ca13a2e5356bc4bf5635c276cc41b46f9f32dfc0f8ddb632006897462: Status 404 returned error can't find the container with id 5609172ca13a2e5356bc4bf5635c276cc41b46f9f32dfc0f8ddb632006897462 Jul 19 15:05:53.204647 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:53.204558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" event={"ID":"8ea3c2dd-c626-48ee-aa71-a35912a689d5","Type":"ContainerStarted","Data":"5609172ca13a2e5356bc4bf5635c276cc41b46f9f32dfc0f8ddb632006897462"} Jul 19 15:05:54.693304 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:54.693260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:56.140479 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.140440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:56.224035 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.224002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" event={"ID":"8ea3c2dd-c626-48ee-aa71-a35912a689d5","Type":"ContainerStarted","Data":"5e26397087a1a22f2686aefb5526eb72c5773e545c8720a1660b9de819324f0a"} Jul 19 15:05:56.224187 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.224116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:05:56.240996 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.240943 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" podStartSLOduration=1.946519033 podStartE2EDuration="4.240926534s" totalImagesPullingTime="2.294407501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:05:56.237862133 +0000 UTC m=+507.102136943" watchObservedRunningTime="2026-07-19 15:05:56.240926534 +0000 UTC m=+507.105201331" Jul 19 15:05:56.933212 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.933177 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk"] Jul 19 15:05:56.940982 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.940956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:05:56.944217 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.944168 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fttpf\"" Jul 19 15:05:56.944357 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.944322 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 19 15:05:56.947576 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:56.947546 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk"] Jul 19 15:05:57.015189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:57.015151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xszk6\" (UniqueName: \"kubernetes.io/projected/7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1-kube-api-access-xszk6\") pod \"dns-operator-controller-manager-864dd78c4c-n9qzk\" (UID: \"7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:05:57.116254 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:57.116223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xszk6\" (UniqueName: \"kubernetes.io/projected/7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1-kube-api-access-xszk6\") pod \"dns-operator-controller-manager-864dd78c4c-n9qzk\" (UID: \"7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:05:57.132335 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:57.132304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xszk6\" (UniqueName: \"kubernetes.io/projected/7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1-kube-api-access-xszk6\") pod \"dns-operator-controller-manager-864dd78c4c-n9qzk\" (UID: \"7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:05:57.252336 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:57.252232 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:05:57.386160 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:57.386134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk"] Jul 19 15:05:57.387865 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:05:57.387833 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3cb68d_15b1_4a19_a72f_e12b74c2b2c1.slice/crio-f74f322bcca3c48f6c8e87aed0c64ee89e9d35f85d81fecf7a46825d5ecf9015 WatchSource:0}: Error finding container f74f322bcca3c48f6c8e87aed0c64ee89e9d35f85d81fecf7a46825d5ecf9015: Status 404 returned error can't find the container with id f74f322bcca3c48f6c8e87aed0c64ee89e9d35f85d81fecf7a46825d5ecf9015 Jul 19 15:05:57.700521 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:57.700491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:05:58.236305 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:05:58.236269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" event={"ID":"7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1","Type":"ContainerStarted","Data":"f74f322bcca3c48f6c8e87aed0c64ee89e9d35f85d81fecf7a46825d5ecf9015"} Jul 19 15:06:00.248661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:00.248619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" event={"ID":"7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1","Type":"ContainerStarted","Data":"b1557054407be8bb53470a3b5db084b06065eb8b8176acb5e94cf5caec2b8b7a"} Jul 19 15:06:00.249050 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:00.248820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:06:00.268391 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:00.268339 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" podStartSLOduration=2.087620516 podStartE2EDuration="4.268321261s" totalImagesPullingTime="2.180700745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:00.265507632 +0000 UTC m=+511.129782430" watchObservedRunningTime="2026-07-19 15:06:00.268321261 +0000 UTC m=+511.132596057" Jul 19 15:06:02.477809 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.477766 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7"] Jul 19 15:06:02.481410 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.481388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.484548 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.484529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qwxx2\"" Jul 19 15:06:02.510996 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.510962 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7"] Jul 19 15:06:02.563190 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.563150 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f03c597-4a82-4a90-9de4-f42b2bed10a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.563190 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.563197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxpd\" (UniqueName: \"kubernetes.io/projected/3f03c597-4a82-4a90-9de4-f42b2bed10a5-kube-api-access-fgxpd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.663987 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.663944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f03c597-4a82-4a90-9de4-f42b2bed10a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.664187 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.663995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxpd\" (UniqueName: \"kubernetes.io/projected/3f03c597-4a82-4a90-9de4-f42b2bed10a5-kube-api-access-fgxpd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.664345 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.664324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f03c597-4a82-4a90-9de4-f42b2bed10a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.671645 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.671624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxpd\" (UniqueName: \"kubernetes.io/projected/3f03c597-4a82-4a90-9de4-f42b2bed10a5-kube-api-access-fgxpd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.791020 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.790929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:02.927371 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:02.927346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7"] Jul 19 15:06:02.929181 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:06:02.929153 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f03c597_4a82_4a90_9de4_f42b2bed10a5.slice/crio-a5cd01ef665cb2a11bbbede6af1b24cde8e83809946ad325c33b36e466f86868 WatchSource:0}: Error finding container a5cd01ef665cb2a11bbbede6af1b24cde8e83809946ad325c33b36e466f86868: Status 404 returned error can't find the container with id a5cd01ef665cb2a11bbbede6af1b24cde8e83809946ad325c33b36e466f86868 Jul 19 15:06:03.265969 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:03.265937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" event={"ID":"3f03c597-4a82-4a90-9de4-f42b2bed10a5","Type":"ContainerStarted","Data":"a5cd01ef665cb2a11bbbede6af1b24cde8e83809946ad325c33b36e466f86868"} Jul 19 15:06:03.901284 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:03.901230 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:04.233979 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:04.233884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:04.335395 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:04.335353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:06.523888 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.523818 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl"] Jul 19 15:06:06.527713 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.527687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:06.529873 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.529849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pgm8k\"" Jul 19 15:06:06.538513 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.538491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl"] Jul 19 15:06:06.539475 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.539395 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:06.600628 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.600560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zm6x\" (UniqueName: \"kubernetes.io/projected/d045c417-1016-4fac-a072-1d2bddad6eda-kube-api-access-6zm6x\") pod \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" (UID: \"d045c417-1016-4fac-a072-1d2bddad6eda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:06.701967 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.701928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zm6x\" (UniqueName: \"kubernetes.io/projected/d045c417-1016-4fac-a072-1d2bddad6eda-kube-api-access-6zm6x\") pod \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" (UID: \"d045c417-1016-4fac-a072-1d2bddad6eda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:06.713320 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.713288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zm6x\" (UniqueName: \"kubernetes.io/projected/d045c417-1016-4fac-a072-1d2bddad6eda-kube-api-access-6zm6x\") pod \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" (UID: \"d045c417-1016-4fac-a072-1d2bddad6eda\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:06.841858 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.841765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:06.985570 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:06.985546 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl"] Jul 19 15:06:06.987615 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:06:06.987571 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd045c417_1016_4fac_a072_1d2bddad6eda.slice/crio-95693b92150986db0a783a2a51b9ad3590e0ef54c912b4386b01bc6bf5009bc1 WatchSource:0}: Error finding container 95693b92150986db0a783a2a51b9ad3590e0ef54c912b4386b01bc6bf5009bc1: Status 404 returned error can't find the container with id 95693b92150986db0a783a2a51b9ad3590e0ef54c912b4386b01bc6bf5009bc1 Jul 19 15:06:07.231528 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:07.231501 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-55mt5" Jul 19 15:06:07.291378 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:07.291331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" event={"ID":"d045c417-1016-4fac-a072-1d2bddad6eda","Type":"ContainerStarted","Data":"95693b92150986db0a783a2a51b9ad3590e0ef54c912b4386b01bc6bf5009bc1"} Jul 19 15:06:08.368378 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:08.368341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:09.293543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.293505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:09.306884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.306847 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" event={"ID":"3f03c597-4a82-4a90-9de4-f42b2bed10a5","Type":"ContainerStarted","Data":"7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43"} Jul 19 15:06:09.307044 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.307008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:09.328745 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.328669 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" podStartSLOduration=1.575008973 podStartE2EDuration="7.328651764s" totalImagesPullingTime="5.753642791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:09.325012411 +0000 UTC m=+520.189287209" watchObservedRunningTime="2026-07-19 15:06:09.328651764 +0000 UTC m=+520.192926561" Jul 19 15:06:09.393262 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.393209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:09.891432 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.891395 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:09.992055 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:09.992019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:10.128020 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:10.127975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:10.314045 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:10.313954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" event={"ID":"d045c417-1016-4fac-a072-1d2bddad6eda","Type":"ContainerStarted","Data":"0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6"} Jul 19 15:06:10.314210 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:10.314052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:10.347094 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:10.347041 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" podStartSLOduration=1.907156657 podStartE2EDuration="4.34702676s" totalImagesPullingTime="2.439870103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:10.342339943 +0000 UTC m=+521.206614741" watchObservedRunningTime="2026-07-19 15:06:10.34702676 +0000 UTC m=+521.211301556" Jul 19 15:06:11.256876 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.256844 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9qzk" Jul 19 15:06:11.728783 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.728748 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7"] Jul 19 15:06:11.734017 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.733994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.736381 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.736361 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 19 15:06:11.736532 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.736512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7rbnp\"" Jul 19 15:06:11.736636 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.736620 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 19 15:06:11.747538 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.747517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7"] Jul 19 15:06:11.760003 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.759762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:11.850717 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.850663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvx9r\" (UniqueName: \"kubernetes.io/projected/7cc7aeb2-3edc-4449-bcae-d94bb77456da-kube-api-access-qvx9r\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.850898 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.850828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7cc7aeb2-3edc-4449-bcae-d94bb77456da-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.850898 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.850867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc7aeb2-3edc-4449-bcae-d94bb77456da-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.946866 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.946833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:11.951997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.951975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7cc7aeb2-3edc-4449-bcae-d94bb77456da-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.952092 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.952018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc7aeb2-3edc-4449-bcae-d94bb77456da-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.952190 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.952137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvx9r\" (UniqueName: \"kubernetes.io/projected/7cc7aeb2-3edc-4449-bcae-d94bb77456da-kube-api-access-qvx9r\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.952673 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.952652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7cc7aeb2-3edc-4449-bcae-d94bb77456da-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.954446 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.954427 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc7aeb2-3edc-4449-bcae-d94bb77456da-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:11.959545 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:11.959521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvx9r\" (UniqueName: \"kubernetes.io/projected/7cc7aeb2-3edc-4449-bcae-d94bb77456da-kube-api-access-qvx9r\") pod \"kuadrant-console-plugin-66b5774985-zbbb7\" (UID: \"7cc7aeb2-3edc-4449-bcae-d94bb77456da\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:12.043061 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:12.042968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" Jul 19 15:06:12.174553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:12.174527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7"] Jul 19 15:06:12.175956 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:06:12.175931 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cc7aeb2_3edc_4449_bcae_d94bb77456da.slice/crio-5c42ef9c7391936a59f05a7d0236e308238d7a93af43767f10ae37cb81134d6c WatchSource:0}: Error finding container 5c42ef9c7391936a59f05a7d0236e308238d7a93af43767f10ae37cb81134d6c: Status 404 returned error can't find the container with id 5c42ef9c7391936a59f05a7d0236e308238d7a93af43767f10ae37cb81134d6c Jul 19 15:06:12.327801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:12.327715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" event={"ID":"7cc7aeb2-3edc-4449-bcae-d94bb77456da","Type":"ContainerStarted","Data":"5c42ef9c7391936a59f05a7d0236e308238d7a93af43767f10ae37cb81134d6c"} Jul 19 15:06:13.901843 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:13.901792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:13.998545 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:13.998502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:18.897858 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:18.897796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:19.008260 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:19.008216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:20.316605 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:20.316559 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:21.323717 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:21.323683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:22.021408 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.021327 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7"] Jul 19 15:06:22.021617 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.021568 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" containerName="manager" containerID="cri-o://7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43" gracePeriod=2 Jul 19 15:06:22.036786 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.036757 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7"] Jul 19 15:06:22.045793 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.045752 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3f03c597-4a82-4a90-9de4-f42b2bed10a5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\": pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:22.050092 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.050063 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl"] Jul 19 15:06:22.050315 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.050286 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" containerName="manager" containerID="cri-o://0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6" gracePeriod=2 Jul 19 15:06:22.052517 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.052482 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:22.054325 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.054301 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg"] Jul 19 15:06:22.054915 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.054894 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" containerName="manager" Jul 19 15:06:22.055054 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.055041 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" containerName="manager" Jul 19 15:06:22.055273 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.055258 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" containerName="manager" Jul 19 15:06:22.061177 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.061160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.066531 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.066505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg"] Jul 19 15:06:22.073664 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.073636 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl"] Jul 19 15:06:22.076504 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.076476 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml"] Jul 19 15:06:22.076954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.076937 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" containerName="manager" Jul 19 15:06:22.077056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.076956 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" containerName="manager" Jul 19 15:06:22.077113 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.077071 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" containerName="manager" Jul 19 15:06:22.079350 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.079315 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d045c417-1016-4fac-a072-1d2bddad6eda\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-vvvhl\": pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:22.080686 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.080667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:22.088906 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.088877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml"] Jul 19 15:06:22.105482 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.105416 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:22.107217 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.107027 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:22.108746 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.108715 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:22.126241 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:22.126201 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:22.250860 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.250822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404c06fe-1541-4f56-b2b0-407cf12da865-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84njg\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.251040 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.250973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcxn\" (UniqueName: \"kubernetes.io/projected/404c06fe-1541-4f56-b2b0-407cf12da865-kube-api-access-8qcxn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84njg\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.251040 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.251022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7j5\" (UniqueName: \"kubernetes.io/projected/0f7f1a0d-78ea-431d-bc88-cfca9e98a75a-kube-api-access-wt7j5\") pod \"limitador-operator-controller-manager-859d7b78bd-rb9ml\" (UID: \"0f7f1a0d-78ea-431d-bc88-cfca9e98a75a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:22.352362 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.352256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404c06fe-1541-4f56-b2b0-407cf12da865-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84njg\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.352810 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.352390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcxn\" (UniqueName: \"kubernetes.io/projected/404c06fe-1541-4f56-b2b0-407cf12da865-kube-api-access-8qcxn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84njg\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.352810 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.352435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7j5\" (UniqueName: \"kubernetes.io/projected/0f7f1a0d-78ea-431d-bc88-cfca9e98a75a-kube-api-access-wt7j5\") pod \"limitador-operator-controller-manager-859d7b78bd-rb9ml\" (UID: \"0f7f1a0d-78ea-431d-bc88-cfca9e98a75a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:22.352810 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.352710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404c06fe-1541-4f56-b2b0-407cf12da865-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84njg\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.363765 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.363733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7j5\" (UniqueName: \"kubernetes.io/projected/0f7f1a0d-78ea-431d-bc88-cfca9e98a75a-kube-api-access-wt7j5\") pod \"limitador-operator-controller-manager-859d7b78bd-rb9ml\" (UID: \"0f7f1a0d-78ea-431d-bc88-cfca9e98a75a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:22.363924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.363736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcxn\" (UniqueName: \"kubernetes.io/projected/404c06fe-1541-4f56-b2b0-407cf12da865-kube-api-access-8qcxn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-84njg\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.439496 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.439454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:22.445605 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:22.445556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:25.071278 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:25.071235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:25.177635 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:25.177577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:28.873936 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:28.873845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:28.974968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:28.974932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:29.075741 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:29.075682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:29.132708 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:29.132609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:30.659177 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:30.659129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:30.712160 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:30.712118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:33.798942 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.798904 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:33.801002 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:33.800965 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:33.817036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.816984 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:33.817441 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.817413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml"] Jul 19 15:06:33.818863 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:33.818685 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:33.820186 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:33.820157 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:33.821289 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:06:33.821265 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7f1a0d_78ea_431d_bc88_cfca9e98a75a.slice/crio-d0c2be2cea0c5782052b3441a4bc3d475bef5257be1a036450cbd825eeb6bda4 WatchSource:0}: Error finding container d0c2be2cea0c5782052b3441a4bc3d475bef5257be1a036450cbd825eeb6bda4: Status 404 returned error can't find the container with id d0c2be2cea0c5782052b3441a4bc3d475bef5257be1a036450cbd825eeb6bda4 Jul 19 15:06:33.850631 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.850606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg"] Jul 19 15:06:33.854796 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:06:33.854767 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404c06fe_1541_4f56_b2b0_407cf12da865.slice/crio-3d56b5f5ab554ed3febc57ced44c7bbafcedc88c540a98ca3b4341dfc6099c32 WatchSource:0}: Error finding container 3d56b5f5ab554ed3febc57ced44c7bbafcedc88c540a98ca3b4341dfc6099c32: Status 404 returned error can't find the container with id 3d56b5f5ab554ed3febc57ced44c7bbafcedc88c540a98ca3b4341dfc6099c32 Jul 19 15:06:33.857867 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.857820 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f03c597-4a82-4a90-9de4-f42b2bed10a5-extensions-socket-volume\") pod \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " Jul 19 15:06:33.857942 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.857880 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zm6x\" (UniqueName: \"kubernetes.io/projected/d045c417-1016-4fac-a072-1d2bddad6eda-kube-api-access-6zm6x\") pod \"d045c417-1016-4fac-a072-1d2bddad6eda\" (UID: \"d045c417-1016-4fac-a072-1d2bddad6eda\") " Jul 19 15:06:33.858414 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.858025 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgxpd\" (UniqueName: \"kubernetes.io/projected/3f03c597-4a82-4a90-9de4-f42b2bed10a5-kube-api-access-fgxpd\") pod \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\" (UID: \"3f03c597-4a82-4a90-9de4-f42b2bed10a5\") " Jul 19 15:06:33.858414 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.858361 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f03c597-4a82-4a90-9de4-f42b2bed10a5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3f03c597-4a82-4a90-9de4-f42b2bed10a5" (UID: "3f03c597-4a82-4a90-9de4-f42b2bed10a5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:06:33.860308 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.860284 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d045c417-1016-4fac-a072-1d2bddad6eda-kube-api-access-6zm6x" (OuterVolumeSpecName: "kube-api-access-6zm6x") pod "d045c417-1016-4fac-a072-1d2bddad6eda" (UID: "d045c417-1016-4fac-a072-1d2bddad6eda"). InnerVolumeSpecName "kube-api-access-6zm6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:06:33.860398 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.860302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f03c597-4a82-4a90-9de4-f42b2bed10a5-kube-api-access-fgxpd" (OuterVolumeSpecName: "kube-api-access-fgxpd") pod "3f03c597-4a82-4a90-9de4-f42b2bed10a5" (UID: "3f03c597-4a82-4a90-9de4-f42b2bed10a5"). InnerVolumeSpecName "kube-api-access-fgxpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:06:33.959661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.959625 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f03c597-4a82-4a90-9de4-f42b2bed10a5-extensions-socket-volume\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:06:33.959661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.959656 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zm6x\" (UniqueName: \"kubernetes.io/projected/d045c417-1016-4fac-a072-1d2bddad6eda-kube-api-access-6zm6x\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:06:33.959661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:33.959665 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgxpd\" (UniqueName: \"kubernetes.io/projected/3f03c597-4a82-4a90-9de4-f42b2bed10a5-kube-api-access-fgxpd\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:06:34.473759 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.473725 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" containerID="7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43" exitCode=0 Jul 19 15:06:34.473957 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.473776 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:34.473957 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.473818 2580 scope.go:117] "RemoveContainer" containerID="7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43" Jul 19 15:06:34.475608 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.475551 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" event={"ID":"7cc7aeb2-3edc-4449-bcae-d94bb77456da","Type":"ContainerStarted","Data":"080d692f05008b7b380d5e43e5be42ef049f63a9838177635a81ec88631bcbf7"} Jul 19 15:06:34.475861 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.475832 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:34.483703 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.477492 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:34.483703 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.479258 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:34.483703 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.480852 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:34.485648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.485476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" event={"ID":"0f7f1a0d-78ea-431d-bc88-cfca9e98a75a","Type":"ContainerStarted","Data":"d859ba6ea4f92f64af7a08e3dd87ad020b6bd8b6c5a73eebbba9d0d0c7d716c4"} Jul 19 15:06:34.485648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.485557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" event={"ID":"0f7f1a0d-78ea-431d-bc88-cfca9e98a75a","Type":"ContainerStarted","Data":"d0c2be2cea0c5782052b3441a4bc3d475bef5257be1a036450cbd825eeb6bda4"} Jul 19 15:06:34.485648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.485641 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:34.487236 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.487216 2580 generic.go:356] "Generic (PLEG): container finished" podID="d045c417-1016-4fac-a072-1d2bddad6eda" containerID="0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6" exitCode=0 Jul 19 15:06:34.487333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.487260 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:34.488872 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.488849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" event={"ID":"404c06fe-1541-4f56-b2b0-407cf12da865","Type":"ContainerStarted","Data":"d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09"} Jul 19 15:06:34.488987 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.488890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" event={"ID":"404c06fe-1541-4f56-b2b0-407cf12da865","Type":"ContainerStarted","Data":"3d56b5f5ab554ed3febc57ced44c7bbafcedc88c540a98ca3b4341dfc6099c32"} Jul 19 15:06:34.488987 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.488936 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:34.491247 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.491234 2580 scope.go:117] "RemoveContainer" containerID="7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43" Jul 19 15:06:34.491475 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.491458 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43\": container with ID starting with 7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43 not found: ID does not exist" containerID="7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43" Jul 19 15:06:34.491538 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.491482 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43"} err="failed to get container status \"7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43\": rpc error: code = NotFound desc = could not find container \"7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43\": container with ID starting with 7fa1211db36fc585406847173b34f10d29438344aa1d0441585085f9cd399c43 not found: ID does not exist" Jul 19 15:06:34.491538 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.491498 2580 scope.go:117] "RemoveContainer" containerID="0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6" Jul 19 15:06:34.498833 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.498793 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zbbb7" podStartSLOduration=1.9783069599999998 podStartE2EDuration="23.498779554s" totalImagesPullingTime="21.520472594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:34.497404828 +0000 UTC m=+545.361679627" watchObservedRunningTime="2026-07-19 15:06:34.498779554 +0000 UTC m=+545.363054357" Jul 19 15:06:34.499182 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.499155 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vvvhl\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vvvhl" Jul 19 15:06:34.500622 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.500565 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zx4w7\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zx4w7" Jul 19 15:06:34.500692 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.500653 2580 scope.go:117] "RemoveContainer" containerID="0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6" Jul 19 15:06:34.500958 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:34.500938 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6\": container with ID starting with 0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6 not found: ID does not exist" containerID="0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6" Jul 19 15:06:34.501018 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.500968 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6"} err="failed to get container status \"0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6\": rpc error: code = NotFound desc = could not find container \"0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6\": container with ID starting with 0a44f3e30c97d9d38ed5a41d3956c809b3688667fde937b6eea1a798a5b2e3e6 not found: ID does not exist" Jul 19 15:06:34.519891 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.519847 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" podStartSLOduration=12.519831484000001 podStartE2EDuration="12.519831484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:34.517462216 +0000 UTC m=+545.381737013" watchObservedRunningTime="2026-07-19 15:06:34.519831484 +0000 UTC m=+545.384106349" Jul 19 15:06:34.534420 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:34.534365 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" podStartSLOduration=12.534349873 podStartE2EDuration="12.534349873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:34.532796804 +0000 UTC m=+545.397071625" watchObservedRunningTime="2026-07-19 15:06:34.534349873 +0000 UTC m=+545.398624669" Jul 19 15:06:35.663162 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:35.663127 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f03c597-4a82-4a90-9de4-f42b2bed10a5" path="/var/lib/kubelet/pods/3f03c597-4a82-4a90-9de4-f42b2bed10a5/volumes" Jul 19 15:06:35.663543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:35.663447 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d045c417-1016-4fac-a072-1d2bddad6eda" path="/var/lib/kubelet/pods/d045c417-1016-4fac-a072-1d2bddad6eda/volumes" Jul 19 15:06:35.818318 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:35.818271 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:35.919454 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:35.919362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:38.134394 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.134352 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg"] Jul 19 15:06:38.134924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.134612 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" podUID="404c06fe-1541-4f56-b2b0-407cf12da865" containerName="manager" containerID="cri-o://d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09" gracePeriod=10 Jul 19 15:06:38.136264 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.136243 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:38.387749 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.387686 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:38.499077 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.499040 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qcxn\" (UniqueName: \"kubernetes.io/projected/404c06fe-1541-4f56-b2b0-407cf12da865-kube-api-access-8qcxn\") pod \"404c06fe-1541-4f56-b2b0-407cf12da865\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " Jul 19 15:06:38.499277 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.499131 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404c06fe-1541-4f56-b2b0-407cf12da865-extensions-socket-volume\") pod \"404c06fe-1541-4f56-b2b0-407cf12da865\" (UID: \"404c06fe-1541-4f56-b2b0-407cf12da865\") " Jul 19 15:06:38.499553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.499520 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404c06fe-1541-4f56-b2b0-407cf12da865-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "404c06fe-1541-4f56-b2b0-407cf12da865" (UID: "404c06fe-1541-4f56-b2b0-407cf12da865"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:06:38.501263 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.501242 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404c06fe-1541-4f56-b2b0-407cf12da865-kube-api-access-8qcxn" (OuterVolumeSpecName: "kube-api-access-8qcxn") pod "404c06fe-1541-4f56-b2b0-407cf12da865" (UID: "404c06fe-1541-4f56-b2b0-407cf12da865"). InnerVolumeSpecName "kube-api-access-8qcxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:06:38.514903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.514871 2580 generic.go:356] "Generic (PLEG): container finished" podID="404c06fe-1541-4f56-b2b0-407cf12da865" containerID="d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09" exitCode=0 Jul 19 15:06:38.515049 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.514914 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" Jul 19 15:06:38.515049 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.514953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" event={"ID":"404c06fe-1541-4f56-b2b0-407cf12da865","Type":"ContainerDied","Data":"d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09"} Jul 19 15:06:38.515049 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.515004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg" event={"ID":"404c06fe-1541-4f56-b2b0-407cf12da865","Type":"ContainerDied","Data":"3d56b5f5ab554ed3febc57ced44c7bbafcedc88c540a98ca3b4341dfc6099c32"} Jul 19 15:06:38.515049 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.515029 2580 scope.go:117] "RemoveContainer" containerID="d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09" Jul 19 15:06:38.524614 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.524563 2580 scope.go:117] "RemoveContainer" containerID="d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09" Jul 19 15:06:38.524908 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:06:38.524889 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09\": container with ID starting with d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09 not found: ID does not exist" containerID="d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09" Jul 19 15:06:38.524967 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.524915 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09"} err="failed to get container status \"d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09\": rpc error: code = NotFound desc = could not find container \"d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09\": container with ID starting with d773e96e74abd43e522e779f0febb5ba41435f0c1743a471fe3138e90e13de09 not found: ID does not exist" Jul 19 15:06:38.550111 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.550078 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg"] Jul 19 15:06:38.554707 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.554679 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-84njg"] Jul 19 15:06:38.600726 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.600694 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcxn\" (UniqueName: \"kubernetes.io/projected/404c06fe-1541-4f56-b2b0-407cf12da865-kube-api-access-8qcxn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:06:38.600726 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:38.600721 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/404c06fe-1541-4f56-b2b0-407cf12da865-extensions-socket-volume\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:06:39.662756 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:39.662718 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404c06fe-1541-4f56-b2b0-407cf12da865" path="/var/lib/kubelet/pods/404c06fe-1541-4f56-b2b0-407cf12da865/volumes" Jul 19 15:06:39.664615 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:39.664569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:39.764694 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:39.764635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:42.844759 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:42.844683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:42.943466 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:42.943432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:45.498328 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:45.498294 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rb9ml" Jul 19 15:06:46.651487 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:46.651446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:46.704391 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:46.704357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:47.056928 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:47.056847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:47.157479 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:47.157425 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:49.664125 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:49.664092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:49.725920 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:49.725857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:54.485559 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.485530 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq"] Jul 19 15:06:54.485970 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.485941 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404c06fe-1541-4f56-b2b0-407cf12da865" containerName="manager" Jul 19 15:06:54.485970 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.485957 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="404c06fe-1541-4f56-b2b0-407cf12da865" containerName="manager" Jul 19 15:06:54.486063 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.486054 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="404c06fe-1541-4f56-b2b0-407cf12da865" containerName="manager" Jul 19 15:06:54.505295 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.505261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:54.576421 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.576381 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq"] Jul 19 15:06:54.576638 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.576527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.578771 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.578747 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dhp6c\"" Jul 19 15:06:54.755884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.755804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.755884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.755849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.755884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.755875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.756128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.755898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.756128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.755974 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.756128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.756036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.756128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.756094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.756128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.756121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.756365 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.756148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrm9\" (UniqueName: \"kubernetes.io/projected/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-kube-api-access-4jrm9\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857040 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857243 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857243 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857243 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857243 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857243 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrm9\" (UniqueName: \"kubernetes.io/projected/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-kube-api-access-4jrm9\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857800 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857800 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.857915 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.857841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.858122 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.858106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.859512 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.859492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.859610 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.859558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.865371 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.865345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.865477 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.865417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrm9\" (UniqueName: \"kubernetes.io/projected/f09ac2ee-ebf4-494e-a842-91be16d9bc6a-kube-api-access-4jrm9\") pod \"maas-default-gateway-openshift-default-56654dcf7-zsgrq\" (UID: \"f09ac2ee-ebf4-494e-a842-91be16d9bc6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.888399 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.888372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:54.901440 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:54.901408 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:55.065534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.065502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:55.100715 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.100686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq"] Jul 19 15:06:55.102103 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:06:55.102078 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09ac2ee_ebf4_494e_a842_91be16d9bc6a.slice/crio-0b870cb56cbcddb4a8f50994ceb4ca8e8d24ac1ea988a6803cb14ca811de721c WatchSource:0}: Error finding container 0b870cb56cbcddb4a8f50994ceb4ca8e8d24ac1ea988a6803cb14ca811de721c: Status 404 returned error can't find the container with id 0b870cb56cbcddb4a8f50994ceb4ca8e8d24ac1ea988a6803cb14ca811de721c Jul 19 15:06:55.104749 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.104717 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:06:55.104819 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.104786 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:06:55.104863 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.104815 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:06:55.208306 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.208273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:06:55.613853 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.613816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" event={"ID":"f09ac2ee-ebf4-494e-a842-91be16d9bc6a","Type":"ContainerStarted","Data":"f55a17d94ce5ec46b766f4d5a60e84f4ba47bea1ce522f7ed1adb148fe3a0f5a"} Jul 19 15:06:55.614266 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.613872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" event={"ID":"f09ac2ee-ebf4-494e-a842-91be16d9bc6a","Type":"ContainerStarted","Data":"0b870cb56cbcddb4a8f50994ceb4ca8e8d24ac1ea988a6803cb14ca811de721c"} Jul 19 15:06:55.634175 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.634129 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" podStartSLOduration=1.6341143310000001 podStartE2EDuration="1.634114331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:06:55.631695132 +0000 UTC m=+566.495969947" watchObservedRunningTime="2026-07-19 15:06:55.634114331 +0000 UTC m=+566.498389127" Jul 19 15:06:55.889425 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:55.889336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:56.894435 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:56.894393 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:57.631657 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:57.631622 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:06:57.632742 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:06:57.632716 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:07:01.664358 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:01.664326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:01.766084 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:01.766046 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:11.871857 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.871819 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:11.874407 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.874377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:11.875061 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.875042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:11.877013 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.876993 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 19 15:07:11.882596 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.882563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:11.895445 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.895421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bd81ba5-24aa-4cee-9497-e9bbd6313295-config-file\") pod \"limitador-limitador-64c94fd64f-gbrfc\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:11.895570 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.895491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7zj\" (UniqueName: \"kubernetes.io/projected/0bd81ba5-24aa-4cee-9497-e9bbd6313295-kube-api-access-pg7zj\") pod \"limitador-limitador-64c94fd64f-gbrfc\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:11.969801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.969768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:11.977672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.977638 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:11.996764 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.996728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bd81ba5-24aa-4cee-9497-e9bbd6313295-config-file\") pod \"limitador-limitador-64c94fd64f-gbrfc\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:11.996896 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.996784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7zj\" (UniqueName: \"kubernetes.io/projected/0bd81ba5-24aa-4cee-9497-e9bbd6313295-kube-api-access-pg7zj\") pod \"limitador-limitador-64c94fd64f-gbrfc\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:11.997326 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:11.997309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bd81ba5-24aa-4cee-9497-e9bbd6313295-config-file\") pod \"limitador-limitador-64c94fd64f-gbrfc\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:12.003739 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.003720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7zj\" (UniqueName: \"kubernetes.io/projected/0bd81ba5-24aa-4cee-9497-e9bbd6313295-kube-api-access-pg7zj\") pod \"limitador-limitador-64c94fd64f-gbrfc\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:12.033924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.033879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:12.187297 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.187213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:12.326593 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.326558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:12.328459 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:07:12.328438 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd81ba5_24aa_4cee_9497_e9bbd6313295.slice/crio-4cf7560f2098b5271bc667ad509703f7952b4894e7993975a846389d16e752b1 WatchSource:0}: Error finding container 4cf7560f2098b5271bc667ad509703f7952b4894e7993975a846389d16e752b1: Status 404 returned error can't find the container with id 4cf7560f2098b5271bc667ad509703f7952b4894e7993975a846389d16e752b1 Jul 19 15:07:12.689977 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.689943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:12.715020 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.714988 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" event={"ID":"0bd81ba5-24aa-4cee-9497-e9bbd6313295","Type":"ContainerStarted","Data":"4cf7560f2098b5271bc667ad509703f7952b4894e7993975a846389d16e752b1"} Jul 19 15:07:12.752359 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.752316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:12.756854 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.756831 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9hwj"] Jul 19 15:07:12.760756 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.760741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:12.762872 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.762853 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-msf75\"" Jul 19 15:07:12.766444 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.766423 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9hwj"] Jul 19 15:07:12.804024 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.803995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7d6s\" (UniqueName: \"kubernetes.io/projected/53c2044e-b261-47a0-8a2f-00e826c027d8-kube-api-access-g7d6s\") pod \"authorino-75d556bd77-f9hwj\" (UID: \"53c2044e-b261-47a0-8a2f-00e826c027d8\") " pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:12.904725 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.904685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7d6s\" (UniqueName: \"kubernetes.io/projected/53c2044e-b261-47a0-8a2f-00e826c027d8-kube-api-access-g7d6s\") pod \"authorino-75d556bd77-f9hwj\" (UID: \"53c2044e-b261-47a0-8a2f-00e826c027d8\") " pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:12.912267 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:12.912239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7d6s\" (UniqueName: \"kubernetes.io/projected/53c2044e-b261-47a0-8a2f-00e826c027d8-kube-api-access-g7d6s\") pod \"authorino-75d556bd77-f9hwj\" (UID: \"53c2044e-b261-47a0-8a2f-00e826c027d8\") " pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:13.072030 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:13.071927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:13.258817 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:13.258616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9hwj"] Jul 19 15:07:13.260823 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:07:13.260774 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53c2044e_b261_47a0_8a2f_00e826c027d8.slice/crio-d12e88224135644873e14bf720016660f0674d4c174e3cabd7794915fa39970c WatchSource:0}: Error finding container d12e88224135644873e14bf720016660f0674d4c174e3cabd7794915fa39970c: Status 404 returned error can't find the container with id d12e88224135644873e14bf720016660f0674d4c174e3cabd7794915fa39970c Jul 19 15:07:13.724704 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:13.724669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9hwj" event={"ID":"53c2044e-b261-47a0-8a2f-00e826c027d8","Type":"ContainerStarted","Data":"d12e88224135644873e14bf720016660f0674d4c174e3cabd7794915fa39970c"} Jul 19 15:07:15.742125 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:15.742046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" event={"ID":"0bd81ba5-24aa-4cee-9497-e9bbd6313295","Type":"ContainerStarted","Data":"4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd"} Jul 19 15:07:15.742570 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:15.742132 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:15.762290 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:15.762229 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" podStartSLOduration=1.935499724 podStartE2EDuration="4.762213698s" totalImagesPullingTime="2.826713974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:15.759899869 +0000 UTC m=+586.624174666" watchObservedRunningTime="2026-07-19 15:07:15.762213698 +0000 UTC m=+586.626488494" Jul 19 15:07:17.756451 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:17.756406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9hwj" event={"ID":"53c2044e-b261-47a0-8a2f-00e826c027d8","Type":"ContainerStarted","Data":"8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2"} Jul 19 15:07:17.771226 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:17.771181 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-f9hwj" podStartSLOduration=2.051602537 podStartE2EDuration="5.771164268s" totalImagesPullingTime="3.719561731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:17.769624437 +0000 UTC m=+588.633899236" watchObservedRunningTime="2026-07-19 15:07:17.771164268 +0000 UTC m=+588.635439065" Jul 19 15:07:26.751354 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:26.751323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:26.828166 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:26.828129 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:26.828388 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:26.828334 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" podUID="0bd81ba5-24aa-4cee-9497-e9bbd6313295" containerName="limitador" containerID="cri-o://4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd" gracePeriod=30 Jul 19 15:07:27.389126 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.389058 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:27.439042 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.439005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg7zj\" (UniqueName: \"kubernetes.io/projected/0bd81ba5-24aa-4cee-9497-e9bbd6313295-kube-api-access-pg7zj\") pod \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " Jul 19 15:07:27.439210 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.439141 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bd81ba5-24aa-4cee-9497-e9bbd6313295-config-file\") pod \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\" (UID: \"0bd81ba5-24aa-4cee-9497-e9bbd6313295\") " Jul 19 15:07:27.439522 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.439499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd81ba5-24aa-4cee-9497-e9bbd6313295-config-file" (OuterVolumeSpecName: "config-file") pod "0bd81ba5-24aa-4cee-9497-e9bbd6313295" (UID: "0bd81ba5-24aa-4cee-9497-e9bbd6313295"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:07:27.441158 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.441132 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd81ba5-24aa-4cee-9497-e9bbd6313295-kube-api-access-pg7zj" (OuterVolumeSpecName: "kube-api-access-pg7zj") pod "0bd81ba5-24aa-4cee-9497-e9bbd6313295" (UID: "0bd81ba5-24aa-4cee-9497-e9bbd6313295"). InnerVolumeSpecName "kube-api-access-pg7zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:07:27.540648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.540559 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0bd81ba5-24aa-4cee-9497-e9bbd6313295-config-file\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:07:27.540648 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.540647 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7zj\" (UniqueName: \"kubernetes.io/projected/0bd81ba5-24aa-4cee-9497-e9bbd6313295-kube-api-access-pg7zj\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:07:27.812458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.812419 2580 generic.go:356] "Generic (PLEG): container finished" podID="0bd81ba5-24aa-4cee-9497-e9bbd6313295" containerID="4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd" exitCode=0 Jul 19 15:07:27.812879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.812469 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" Jul 19 15:07:27.812879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.812493 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" event={"ID":"0bd81ba5-24aa-4cee-9497-e9bbd6313295","Type":"ContainerDied","Data":"4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd"} Jul 19 15:07:27.812879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.812545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gbrfc" event={"ID":"0bd81ba5-24aa-4cee-9497-e9bbd6313295","Type":"ContainerDied","Data":"4cf7560f2098b5271bc667ad509703f7952b4894e7993975a846389d16e752b1"} Jul 19 15:07:27.812879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.812573 2580 scope.go:117] "RemoveContainer" containerID="4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd" Jul 19 15:07:27.821613 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.821594 2580 scope.go:117] "RemoveContainer" containerID="4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd" Jul 19 15:07:27.821901 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:07:27.821873 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd\": container with ID starting with 4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd not found: ID does not exist" containerID="4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd" Jul 19 15:07:27.821990 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.821910 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd"} err="failed to get container status \"4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd\": rpc error: code = NotFound desc = could not find container \"4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd\": container with ID starting with 4d75e85183e798ac283c554a36398ffe9d9f703281ffce0159c0b9bbaeea8cbd not found: ID does not exist" Jul 19 15:07:27.828461 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.828438 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:27.832225 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:27.832206 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gbrfc"] Jul 19 15:07:28.089640 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.089532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:28.187845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.187814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:28.539821 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.539780 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk"] Jul 19 15:07:28.540309 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.540284 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd81ba5-24aa-4cee-9497-e9bbd6313295" containerName="limitador" Jul 19 15:07:28.540484 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.540465 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd81ba5-24aa-4cee-9497-e9bbd6313295" containerName="limitador" Jul 19 15:07:28.540660 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.540646 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd81ba5-24aa-4cee-9497-e9bbd6313295" containerName="limitador" Jul 19 15:07:28.545227 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.545200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.548056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.547814 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 15:07:28.548056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.547835 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:07:28.548056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.547852 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 19 15:07:28.548056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.547997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zwrzj\"" Jul 19 15:07:28.550706 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.550685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk"] Jul 19 15:07:28.566776 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.566752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:28.650474 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.650445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/860e4b9e-5c4f-4d58-82db-0bab17c14c92-data\") pod \"postgres-69fb4594cf-4f5nk\" (UID: \"860e4b9e-5c4f-4d58-82db-0bab17c14c92\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.650692 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.650505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgls9\" (UniqueName: \"kubernetes.io/projected/860e4b9e-5c4f-4d58-82db-0bab17c14c92-kube-api-access-cgls9\") pod \"postgres-69fb4594cf-4f5nk\" (UID: \"860e4b9e-5c4f-4d58-82db-0bab17c14c92\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.751674 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.751633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgls9\" (UniqueName: \"kubernetes.io/projected/860e4b9e-5c4f-4d58-82db-0bab17c14c92-kube-api-access-cgls9\") pod \"postgres-69fb4594cf-4f5nk\" (UID: \"860e4b9e-5c4f-4d58-82db-0bab17c14c92\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.751868 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.751789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/860e4b9e-5c4f-4d58-82db-0bab17c14c92-data\") pod \"postgres-69fb4594cf-4f5nk\" (UID: \"860e4b9e-5c4f-4d58-82db-0bab17c14c92\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.752207 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.752182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/860e4b9e-5c4f-4d58-82db-0bab17c14c92-data\") pod \"postgres-69fb4594cf-4f5nk\" (UID: \"860e4b9e-5c4f-4d58-82db-0bab17c14c92\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.761649 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.761625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgls9\" (UniqueName: \"kubernetes.io/projected/860e4b9e-5c4f-4d58-82db-0bab17c14c92-kube-api-access-cgls9\") pod \"postgres-69fb4594cf-4f5nk\" (UID: \"860e4b9e-5c4f-4d58-82db-0bab17c14c92\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.857801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.857774 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:28.995625 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:28.995597 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk"] Jul 19 15:07:28.997393 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:07:28.997364 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860e4b9e_5c4f_4d58_82db_0bab17c14c92.slice/crio-36a567f28e953e9de09442cba263e986b5057e73533a3ad4cbf3445627c8a6e5 WatchSource:0}: Error finding container 36a567f28e953e9de09442cba263e986b5057e73533a3ad4cbf3445627c8a6e5: Status 404 returned error can't find the container with id 36a567f28e953e9de09442cba263e986b5057e73533a3ad4cbf3445627c8a6e5 Jul 19 15:07:29.600366 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:29.600340 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:07:29.600366 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:29.600351 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:07:29.606396 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:29.606364 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:07:29.606548 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:29.606509 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:07:29.663257 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:29.663226 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd81ba5-24aa-4cee-9497-e9bbd6313295" path="/var/lib/kubelet/pods/0bd81ba5-24aa-4cee-9497-e9bbd6313295/volumes" Jul 19 15:07:29.829140 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:29.828952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" event={"ID":"860e4b9e-5c4f-4d58-82db-0bab17c14c92","Type":"ContainerStarted","Data":"36a567f28e953e9de09442cba263e986b5057e73533a3ad4cbf3445627c8a6e5"} Jul 19 15:07:33.749236 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:33.749212 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 19 15:07:33.855620 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:33.855560 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" event={"ID":"860e4b9e-5c4f-4d58-82db-0bab17c14c92","Type":"ContainerStarted","Data":"15bfa5ae21df04b3064a5d5be9a0baaedc055ae0089d9ba929363ca74106009c"} Jul 19 15:07:33.855796 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:33.855725 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:33.870089 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:33.870034 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" podStartSLOduration=1.122015452 podStartE2EDuration="5.870019091s" totalImagesPullingTime="4.748003639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:33.868749765 +0000 UTC m=+604.733024562" watchObservedRunningTime="2026-07-19 15:07:33.870019091 +0000 UTC m=+604.734293926" Jul 19 15:07:39.892322 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:39.892296 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f5nk" Jul 19 15:07:40.202046 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:40.201963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:40.542048 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:40.541967 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:42.480115 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.480077 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9hwj"] Jul 19 15:07:42.480521 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.480247 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-f9hwj" podUID="53c2044e-b261-47a0-8a2f-00e826c027d8" containerName="authorino" containerID="cri-o://8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2" gracePeriod=30 Jul 19 15:07:42.751969 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.751942 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:42.882221 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.882189 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7d6s\" (UniqueName: \"kubernetes.io/projected/53c2044e-b261-47a0-8a2f-00e826c027d8-kube-api-access-g7d6s\") pod \"53c2044e-b261-47a0-8a2f-00e826c027d8\" (UID: \"53c2044e-b261-47a0-8a2f-00e826c027d8\") " Jul 19 15:07:42.884305 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.884280 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c2044e-b261-47a0-8a2f-00e826c027d8-kube-api-access-g7d6s" (OuterVolumeSpecName: "kube-api-access-g7d6s") pod "53c2044e-b261-47a0-8a2f-00e826c027d8" (UID: "53c2044e-b261-47a0-8a2f-00e826c027d8"). InnerVolumeSpecName "kube-api-access-g7d6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:07:42.906192 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.906161 2580 generic.go:356] "Generic (PLEG): container finished" podID="53c2044e-b261-47a0-8a2f-00e826c027d8" containerID="8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2" exitCode=0 Jul 19 15:07:42.906361 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.906203 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9hwj" Jul 19 15:07:42.906361 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.906228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9hwj" event={"ID":"53c2044e-b261-47a0-8a2f-00e826c027d8","Type":"ContainerDied","Data":"8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2"} Jul 19 15:07:42.906361 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.906261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9hwj" event={"ID":"53c2044e-b261-47a0-8a2f-00e826c027d8","Type":"ContainerDied","Data":"d12e88224135644873e14bf720016660f0674d4c174e3cabd7794915fa39970c"} Jul 19 15:07:42.906361 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.906277 2580 scope.go:117] "RemoveContainer" containerID="8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2" Jul 19 15:07:42.915820 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.915803 2580 scope.go:117] "RemoveContainer" containerID="8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2" Jul 19 15:07:42.916156 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:07:42.916075 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2\": container with ID starting with 8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2 not found: ID does not exist" containerID="8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2" Jul 19 15:07:42.916213 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.916167 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2"} err="failed to get container status \"8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2\": rpc error: code = NotFound desc = could not find container \"8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2\": container with ID starting with 8d4fbd9413d1c165bdbc63af622b1c0abf7144c4d5d00e70d4cafd3a7b8066d2 not found: ID does not exist" Jul 19 15:07:42.931143 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.931107 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9hwj"] Jul 19 15:07:42.933337 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.933316 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9hwj"] Jul 19 15:07:42.983747 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:42.983707 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7d6s\" (UniqueName: \"kubernetes.io/projected/53c2044e-b261-47a0-8a2f-00e826c027d8-kube-api-access-g7d6s\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:07:43.662397 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:43.662363 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c2044e-b261-47a0-8a2f-00e826c027d8" path="/var/lib/kubelet/pods/53c2044e-b261-47a0-8a2f-00e826c027d8/volumes" Jul 19 15:07:47.286721 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.286688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:47.298505 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.298476 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586bbb4654-9sgdt"] Jul 19 15:07:47.298903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.298891 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c2044e-b261-47a0-8a2f-00e826c027d8" containerName="authorino" Jul 19 15:07:47.298960 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.298906 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c2044e-b261-47a0-8a2f-00e826c027d8" containerName="authorino" Jul 19 15:07:47.298999 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.298962 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c2044e-b261-47a0-8a2f-00e826c027d8" containerName="authorino" Jul 19 15:07:47.546841 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.546759 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586bbb4654-9sgdt"] Jul 19 15:07:47.546841 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.546799 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-j2fsc"] Jul 19 15:07:47.547022 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.546874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.549919 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.549876 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dtfm\"" Jul 19 15:07:47.549919 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.549879 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 19 15:07:47.550106 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.549916 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 19 15:07:47.574570 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.574533 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-j2fsc"] Jul 19 15:07:47.574750 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.574650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.724747 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.724705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16efbd1a-a6c8-4110-a1bf-f914c672092d-webhook-cert\") pod \"maas-controller-6cb557cbc4-j2fsc\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.724932 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.724816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f493402-4902-48b8-bfa3-e49b135c46e4-webhook-cert\") pod \"maas-controller-586bbb4654-9sgdt\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.724932 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.724850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqsx\" (UniqueName: \"kubernetes.io/projected/9f493402-4902-48b8-bfa3-e49b135c46e4-kube-api-access-4nqsx\") pod \"maas-controller-586bbb4654-9sgdt\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.724932 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.724883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqhg\" (UniqueName: \"kubernetes.io/projected/16efbd1a-a6c8-4110-a1bf-f914c672092d-kube-api-access-4nqhg\") pod \"maas-controller-6cb557cbc4-j2fsc\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.826043 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.825950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqhg\" (UniqueName: \"kubernetes.io/projected/16efbd1a-a6c8-4110-a1bf-f914c672092d-kube-api-access-4nqhg\") pod \"maas-controller-6cb557cbc4-j2fsc\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.826224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.826086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16efbd1a-a6c8-4110-a1bf-f914c672092d-webhook-cert\") pod \"maas-controller-6cb557cbc4-j2fsc\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.826224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.826142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f493402-4902-48b8-bfa3-e49b135c46e4-webhook-cert\") pod \"maas-controller-586bbb4654-9sgdt\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.826224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.826163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqsx\" (UniqueName: \"kubernetes.io/projected/9f493402-4902-48b8-bfa3-e49b135c46e4-kube-api-access-4nqsx\") pod \"maas-controller-586bbb4654-9sgdt\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.828657 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.828633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f493402-4902-48b8-bfa3-e49b135c46e4-webhook-cert\") pod \"maas-controller-586bbb4654-9sgdt\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.828779 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.828673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16efbd1a-a6c8-4110-a1bf-f914c672092d-webhook-cert\") pod \"maas-controller-6cb557cbc4-j2fsc\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.834015 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.833988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqsx\" (UniqueName: \"kubernetes.io/projected/9f493402-4902-48b8-bfa3-e49b135c46e4-kube-api-access-4nqsx\") pod \"maas-controller-586bbb4654-9sgdt\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.834120 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.834070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqhg\" (UniqueName: \"kubernetes.io/projected/16efbd1a-a6c8-4110-a1bf-f914c672092d-kube-api-access-4nqhg\") pod \"maas-controller-6cb557cbc4-j2fsc\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:47.857645 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.857618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:47.884382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:47.884357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:48.025975 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:48.025943 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586bbb4654-9sgdt"] Jul 19 15:07:48.028059 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:07:48.028015 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f493402_4902_48b8_bfa3_e49b135c46e4.slice/crio-4c5abcb2f36279df854f8c8b1b20d9d6da262641de01f75f8aaa90327fb3f916 WatchSource:0}: Error finding container 4c5abcb2f36279df854f8c8b1b20d9d6da262641de01f75f8aaa90327fb3f916: Status 404 returned error can't find the container with id 4c5abcb2f36279df854f8c8b1b20d9d6da262641de01f75f8aaa90327fb3f916 Jul 19 15:07:48.029650 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:48.029631 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:07:48.049837 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:48.049816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-j2fsc"] Jul 19 15:07:48.052231 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:07:48.052211 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16efbd1a_a6c8_4110_a1bf_f914c672092d.slice/crio-4ebb7cd69b7aedb4f700b0c44d5d0f7891f6c2cebe9b505c783816f300420bab WatchSource:0}: Error finding container 4ebb7cd69b7aedb4f700b0c44d5d0f7891f6c2cebe9b505c783816f300420bab: Status 404 returned error can't find the container with id 4ebb7cd69b7aedb4f700b0c44d5d0f7891f6c2cebe9b505c783816f300420bab Jul 19 15:07:48.332858 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:48.332822 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:48.948655 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:48.948621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586bbb4654-9sgdt" event={"ID":"9f493402-4902-48b8-bfa3-e49b135c46e4","Type":"ContainerStarted","Data":"4c5abcb2f36279df854f8c8b1b20d9d6da262641de01f75f8aaa90327fb3f916"} Jul 19 15:07:48.951220 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:48.950950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" event={"ID":"16efbd1a-a6c8-4110-a1bf-f914c672092d","Type":"ContainerStarted","Data":"4ebb7cd69b7aedb4f700b0c44d5d0f7891f6c2cebe9b505c783816f300420bab"} Jul 19 15:07:50.650109 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.650081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:50.749889 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.749796 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:07:50.963476 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.963438 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586bbb4654-9sgdt" event={"ID":"9f493402-4902-48b8-bfa3-e49b135c46e4","Type":"ContainerStarted","Data":"2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc"} Jul 19 15:07:50.963659 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.963503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:07:50.965131 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.965102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" event={"ID":"16efbd1a-a6c8-4110-a1bf-f914c672092d","Type":"ContainerStarted","Data":"c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0"} Jul 19 15:07:50.965249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.965147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:07:50.982551 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.982506 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-586bbb4654-9sgdt" podStartSLOduration=1.537061132 podStartE2EDuration="3.982491675s" totalImagesPullingTime="2.445430543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:50.981264293 +0000 UTC m=+621.845539087" watchObservedRunningTime="2026-07-19 15:07:50.982491675 +0000 UTC m=+621.846766472" Jul 19 15:07:50.998220 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:07:50.998175 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" podStartSLOduration=1.563418103 podStartE2EDuration="3.998164851s" totalImagesPullingTime="2.434746748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:50.996414632 +0000 UTC m=+621.860689429" watchObservedRunningTime="2026-07-19 15:07:50.998164851 +0000 UTC m=+621.862439702" Jul 19 15:08:01.973941 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:01.973858 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:08:01.973941 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:01.973930 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:08:02.033845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.033813 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586bbb4654-9sgdt"] Jul 19 15:08:02.034040 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.033992 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-586bbb4654-9sgdt" podUID="9f493402-4902-48b8-bfa3-e49b135c46e4" containerName="manager" containerID="cri-o://2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc" gracePeriod=30 Jul 19 15:08:02.291742 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.291717 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:08:02.342565 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.342533 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqsx\" (UniqueName: \"kubernetes.io/projected/9f493402-4902-48b8-bfa3-e49b135c46e4-kube-api-access-4nqsx\") pod \"9f493402-4902-48b8-bfa3-e49b135c46e4\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " Jul 19 15:08:02.342748 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.342649 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f493402-4902-48b8-bfa3-e49b135c46e4-webhook-cert\") pod \"9f493402-4902-48b8-bfa3-e49b135c46e4\" (UID: \"9f493402-4902-48b8-bfa3-e49b135c46e4\") " Jul 19 15:08:02.344773 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.344740 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f493402-4902-48b8-bfa3-e49b135c46e4-kube-api-access-4nqsx" (OuterVolumeSpecName: "kube-api-access-4nqsx") pod "9f493402-4902-48b8-bfa3-e49b135c46e4" (UID: "9f493402-4902-48b8-bfa3-e49b135c46e4"). InnerVolumeSpecName "kube-api-access-4nqsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:08:02.344893 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.344772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f493402-4902-48b8-bfa3-e49b135c46e4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9f493402-4902-48b8-bfa3-e49b135c46e4" (UID: "9f493402-4902-48b8-bfa3-e49b135c46e4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:08:02.443628 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.443568 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f493402-4902-48b8-bfa3-e49b135c46e4-webhook-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:08:02.443770 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:02.443646 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqsx\" (UniqueName: \"kubernetes.io/projected/9f493402-4902-48b8-bfa3-e49b135c46e4-kube-api-access-4nqsx\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:08:03.033392 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.033356 2580 generic.go:356] "Generic (PLEG): container finished" podID="9f493402-4902-48b8-bfa3-e49b135c46e4" containerID="2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc" exitCode=0 Jul 19 15:08:03.033800 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.033399 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586bbb4654-9sgdt" Jul 19 15:08:03.033800 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.033440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586bbb4654-9sgdt" event={"ID":"9f493402-4902-48b8-bfa3-e49b135c46e4","Type":"ContainerDied","Data":"2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc"} Jul 19 15:08:03.033800 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.033493 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586bbb4654-9sgdt" event={"ID":"9f493402-4902-48b8-bfa3-e49b135c46e4","Type":"ContainerDied","Data":"4c5abcb2f36279df854f8c8b1b20d9d6da262641de01f75f8aaa90327fb3f916"} Jul 19 15:08:03.033800 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.033514 2580 scope.go:117] "RemoveContainer" containerID="2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc" Jul 19 15:08:03.042794 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.042777 2580 scope.go:117] "RemoveContainer" containerID="2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc" Jul 19 15:08:03.043028 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:08:03.043013 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc\": container with ID starting with 2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc not found: ID does not exist" containerID="2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc" Jul 19 15:08:03.043072 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.043037 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc"} err="failed to get container status \"2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc\": rpc error: code = NotFound desc = could not find container \"2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc\": container with ID starting with 2bf8dbb8350f07e6b3e374ef1379d7848c23176c803586ace42967c26920cabc not found: ID does not exist" Jul 19 15:08:03.054109 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.054087 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586bbb4654-9sgdt"] Jul 19 15:08:03.057522 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.057503 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-586bbb4654-9sgdt"] Jul 19 15:08:03.662785 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:03.662756 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f493402-4902-48b8-bfa3-e49b135c46e4" path="/var/lib/kubelet/pods/9f493402-4902-48b8-bfa3-e49b135c46e4/volumes" Jul 19 15:08:22.462791 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:22.462750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:22.563522 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:22.563482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:23.044146 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.044109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:23.101019 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.100987 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv"] Jul 19 15:08:23.101384 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.101373 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f493402-4902-48b8-bfa3-e49b135c46e4" containerName="manager" Jul 19 15:08:23.101429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.101386 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f493402-4902-48b8-bfa3-e49b135c46e4" containerName="manager" Jul 19 15:08:23.101478 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.101469 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f493402-4902-48b8-bfa3-e49b135c46e4" containerName="manager" Jul 19 15:08:23.105942 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.105926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.107992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.107959 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 19 15:08:23.108094 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.108074 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ljt6w\"" Jul 19 15:08:23.115178 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.115154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv"] Jul 19 15:08:23.230377 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.230339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:23.239483 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.239452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521464ff-46c0-40f1-bf39-a41c6d5d14cd-maas-api-tls\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.239621 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.239498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521464ff-46c0-40f1-bf39-a41c6d5d14cd-openshift-service-ca\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.239621 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.239555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q4t9\" (UniqueName: \"kubernetes.io/projected/521464ff-46c0-40f1-bf39-a41c6d5d14cd-kube-api-access-4q4t9\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.341027 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.340952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q4t9\" (UniqueName: \"kubernetes.io/projected/521464ff-46c0-40f1-bf39-a41c6d5d14cd-kube-api-access-4q4t9\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.341194 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.341064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521464ff-46c0-40f1-bf39-a41c6d5d14cd-maas-api-tls\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.341194 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.341101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521464ff-46c0-40f1-bf39-a41c6d5d14cd-openshift-service-ca\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.341774 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.341749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/521464ff-46c0-40f1-bf39-a41c6d5d14cd-openshift-service-ca\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.343636 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.343614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/521464ff-46c0-40f1-bf39-a41c6d5d14cd-maas-api-tls\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.353512 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.353487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q4t9\" (UniqueName: \"kubernetes.io/projected/521464ff-46c0-40f1-bf39-a41c6d5d14cd-kube-api-access-4q4t9\") pod \"maas-api-6db48d95f-j8pjv\" (UID: \"521464ff-46c0-40f1-bf39-a41c6d5d14cd\") " pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.416645 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.416613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:23.501679 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.501648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:23.565351 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.565323 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv"] Jul 19 15:08:23.566496 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.566470 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:23.567162 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:08:23.567124 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521464ff_46c0_40f1_bf39_a41c6d5d14cd.slice/crio-5be4f429f606805418cce580682eb890dc19295a2dcf9f990a200cfa0bb27dd0 WatchSource:0}: Error finding container 5be4f429f606805418cce580682eb890dc19295a2dcf9f990a200cfa0bb27dd0: Status 404 returned error can't find the container with id 5be4f429f606805418cce580682eb890dc19295a2dcf9f990a200cfa0bb27dd0 Jul 19 15:08:23.634263 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.634239 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-czp9q"] Jul 19 15:08:23.638853 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.638837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.641233 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.641214 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 19 15:08:23.641337 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.641255 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xkmfd\"" Jul 19 15:08:23.646961 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.646933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-czp9q"] Jul 19 15:08:23.744938 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.744905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df93acff-04ce-4151-b643-68ce4023c5b0-plugins-config-volume\") pod \"payload-processing-5bf47cd988-czp9q\" (UID: \"df93acff-04ce-4151-b643-68ce4023c5b0\") " pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.745138 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.745027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcq7b\" (UniqueName: \"kubernetes.io/projected/df93acff-04ce-4151-b643-68ce4023c5b0-kube-api-access-bcq7b\") pod \"payload-processing-5bf47cd988-czp9q\" (UID: \"df93acff-04ce-4151-b643-68ce4023c5b0\") " pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.846028 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.845987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcq7b\" (UniqueName: \"kubernetes.io/projected/df93acff-04ce-4151-b643-68ce4023c5b0-kube-api-access-bcq7b\") pod \"payload-processing-5bf47cd988-czp9q\" (UID: \"df93acff-04ce-4151-b643-68ce4023c5b0\") " pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.846249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.846060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df93acff-04ce-4151-b643-68ce4023c5b0-plugins-config-volume\") pod \"payload-processing-5bf47cd988-czp9q\" (UID: \"df93acff-04ce-4151-b643-68ce4023c5b0\") " pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.846797 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.846771 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df93acff-04ce-4151-b643-68ce4023c5b0-plugins-config-volume\") pod \"payload-processing-5bf47cd988-czp9q\" (UID: \"df93acff-04ce-4151-b643-68ce4023c5b0\") " pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.854175 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.854147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcq7b\" (UniqueName: \"kubernetes.io/projected/df93acff-04ce-4151-b643-68ce4023c5b0-kube-api-access-bcq7b\") pod \"payload-processing-5bf47cd988-czp9q\" (UID: \"df93acff-04ce-4151-b643-68ce4023c5b0\") " pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:23.952397 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:23.952309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:24.090742 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:24.090711 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-czp9q"] Jul 19 15:08:24.091998 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:08:24.091964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf93acff_04ce_4151_b643_68ce4023c5b0.slice/crio-6db5739c74b571d6befad02a4d3538f04e170de0f4b1eefb978f979b5764486a WatchSource:0}: Error finding container 6db5739c74b571d6befad02a4d3538f04e170de0f4b1eefb978f979b5764486a: Status 404 returned error can't find the container with id 6db5739c74b571d6befad02a4d3538f04e170de0f4b1eefb978f979b5764486a Jul 19 15:08:24.155132 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:24.155092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" event={"ID":"521464ff-46c0-40f1-bf39-a41c6d5d14cd","Type":"ContainerStarted","Data":"5be4f429f606805418cce580682eb890dc19295a2dcf9f990a200cfa0bb27dd0"} Jul 19 15:08:24.157020 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:24.156980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" event={"ID":"df93acff-04ce-4151-b643-68ce4023c5b0","Type":"ContainerStarted","Data":"6db5739c74b571d6befad02a4d3538f04e170de0f4b1eefb978f979b5764486a"} Jul 19 15:08:26.182269 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:26.182232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" event={"ID":"df93acff-04ce-4151-b643-68ce4023c5b0","Type":"ContainerStarted","Data":"d3d4a68419619c883f511fc8055b21592fdadc46aaf9afb59ffd26c1e5d3d17a"} Jul 19 15:08:26.182686 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:26.182313 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:26.183884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:26.183853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" event={"ID":"521464ff-46c0-40f1-bf39-a41c6d5d14cd","Type":"ContainerStarted","Data":"e9badff17c865be5cb50a0fe2d09cb6c61ddc418f00bb3d07c22e9dd510f7b1d"} Jul 19 15:08:26.184000 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:26.183986 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:26.199459 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:26.199399 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" podStartSLOduration=1.199569622 podStartE2EDuration="3.199381724s" totalImagesPullingTime="1.999812102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:08:26.196013074 +0000 UTC m=+657.060287881" watchObservedRunningTime="2026-07-19 15:08:26.199381724 +0000 UTC m=+657.063656522" Jul 19 15:08:26.211888 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:26.211843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" podStartSLOduration=1.4674807460000001 podStartE2EDuration="3.211828412s" totalImagesPullingTime="1.744347666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:08:26.209353823 +0000 UTC m=+657.073628621" watchObservedRunningTime="2026-07-19 15:08:26.211828412 +0000 UTC m=+657.076103209" Jul 19 15:08:31.216411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:31.216324 2580 generic.go:356] "Generic (PLEG): container finished" podID="f09ac2ee-ebf4-494e-a842-91be16d9bc6a" containerID="f55a17d94ce5ec46b766f4d5a60e84f4ba47bea1ce522f7ed1adb148fe3a0f5a" exitCode=0 Jul 19 15:08:31.216411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:31.216396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" event={"ID":"f09ac2ee-ebf4-494e-a842-91be16d9bc6a","Type":"ContainerDied","Data":"f55a17d94ce5ec46b766f4d5a60e84f4ba47bea1ce522f7ed1adb148fe3a0f5a"} Jul 19 15:08:31.216909 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:31.216821 2580 scope.go:117] "RemoveContainer" containerID="f55a17d94ce5ec46b766f4d5a60e84f4ba47bea1ce522f7ed1adb148fe3a0f5a" Jul 19 15:08:31.217871 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:31.217842 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:08:31.217958 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:31.217936 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:08:31.218005 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:31.217986 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 19 15:08:32.195697 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:32.195669 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6db48d95f-j8pjv" Jul 19 15:08:32.225127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:32.225093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" event={"ID":"f09ac2ee-ebf4-494e-a842-91be16d9bc6a","Type":"ContainerStarted","Data":"b5da29d2f3fa059db4ff488a750f5a7d58d5e5e952014fbdd2f9026436d36f7b"} Jul 19 15:08:32.889267 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:32.889230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:08:32.889543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:32.889518 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-zsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 19 15:08:32.889616 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:32.889570 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" podUID="f09ac2ee-ebf4-494e-a842-91be16d9bc6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 19 15:08:33.828773 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:33.828734 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:33.889816 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:33.889780 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-zsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 19 15:08:33.889973 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:33.889840 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" podUID="f09ac2ee-ebf4-494e-a842-91be16d9bc6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 19 15:08:33.935939 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:33.935903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:34.889466 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:34.889428 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-zsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 19 15:08:34.889957 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:34.889491 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" podUID="f09ac2ee-ebf4-494e-a842-91be16d9bc6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 19 15:08:35.889815 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:35.889777 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-zsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 19 15:08:35.890291 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:35.889841 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" podUID="f09ac2ee-ebf4-494e-a842-91be16d9bc6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 19 15:08:36.893372 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:36.893341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:08:36.893760 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:36.893726 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:08:36.894401 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:36.894381 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-zsgrq" Jul 19 15:08:37.192528 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:37.192441 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-czp9q" Jul 19 15:08:45.487509 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.487472 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn"] Jul 19 15:08:45.493037 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.493010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.495224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.495202 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 19 15:08:45.495878 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.495858 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c7xpt\"" Jul 19 15:08:45.495963 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.495905 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 19 15:08:45.496008 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.495948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 19 15:08:45.501276 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.501249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn"] Jul 19 15:08:45.523702 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.523659 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:45.566770 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.566735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.566770 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.566773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991ded96-c855-43d8-a32f-c53290de37ec-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.566997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.566796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.566997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.566814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.566997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.566856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnn9d\" (UniqueName: \"kubernetes.io/projected/991ded96-c855-43d8-a32f-c53290de37ec-kube-api-access-xnn9d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.566997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.566937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.667705 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.667659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.667705 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.667710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991ded96-c855-43d8-a32f-c53290de37ec-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.667954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.667742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.667954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.667766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.667954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.667788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnn9d\" (UniqueName: \"kubernetes.io/projected/991ded96-c855-43d8-a32f-c53290de37ec-kube-api-access-xnn9d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.667954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.667849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.668168 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.668078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.668168 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.668112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.668285 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.668263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.670056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.670036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991ded96-c855-43d8-a32f-c53290de37ec-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.670329 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.670310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991ded96-c855-43d8-a32f-c53290de37ec-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.675007 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.674985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnn9d\" (UniqueName: \"kubernetes.io/projected/991ded96-c855-43d8-a32f-c53290de37ec-kube-api-access-xnn9d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn\" (UID: \"991ded96-c855-43d8-a32f-c53290de37ec\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.807708 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.807612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:45.951591 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:45.951534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn"] Jul 19 15:08:45.952699 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:08:45.952669 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991ded96_c855_43d8_a32f_c53290de37ec.slice/crio-ac7083a558efde2df6f1fb917d436ac20d720aac2f721f2519b11cb3928dc508 WatchSource:0}: Error finding container ac7083a558efde2df6f1fb917d436ac20d720aac2f721f2519b11cb3928dc508: Status 404 returned error can't find the container with id ac7083a558efde2df6f1fb917d436ac20d720aac2f721f2519b11cb3928dc508 Jul 19 15:08:46.309162 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:46.309044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" event={"ID":"991ded96-c855-43d8-a32f-c53290de37ec","Type":"ContainerStarted","Data":"ac7083a558efde2df6f1fb917d436ac20d720aac2f721f2519b11cb3928dc508"} Jul 19 15:08:47.316377 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:47.316338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" event={"ID":"991ded96-c855-43d8-a32f-c53290de37ec","Type":"ContainerStarted","Data":"c744a423f3fd076fc83c2122d145ab993f9fbf893fc084413e0ad9723bb79206"} Jul 19 15:08:47.334058 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:47.334005 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" podStartSLOduration=1.172492379 podStartE2EDuration="2.33398655s" totalImagesPullingTime="1.161494171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:08:47.332326498 +0000 UTC m=+678.196601296" watchObservedRunningTime="2026-07-19 15:08:47.33398655 +0000 UTC m=+678.198261380" Jul 19 15:08:49.365645 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:49.365607 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:49.416114 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:49.416081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:55.433067 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:55.433031 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:08:55.807969 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:55.807877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:55.807969 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:55.807960 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:55.820560 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:55.820531 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:08:56.379117 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:08:56.379090 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn" Jul 19 15:09:16.041715 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:16.041678 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:09:25.199656 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.199539 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6"] Jul 19 15:09:25.203474 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.203455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.205724 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.205703 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 19 15:09:25.213284 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.213261 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6"] Jul 19 15:09:25.224302 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.224250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:09:25.234298 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.234266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.234413 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.234303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpg9\" (UniqueName: \"kubernetes.io/projected/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-kube-api-access-jcpg9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.234413 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.234333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.234413 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.234368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.234413 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.234402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.234557 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.234418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.335659 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.335620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.335822 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.335672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.335822 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.335704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.335822 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.335720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.335822 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.335812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.336011 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.335828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpg9\" (UniqueName: \"kubernetes.io/projected/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-kube-api-access-jcpg9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.336078 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.336055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.336163 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.336141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.336224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.336185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.338026 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.338006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.338309 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.338290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.343749 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.343726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpg9\" (UniqueName: \"kubernetes.io/projected/2d871e4c-674f-43fb-8c9e-5d57913f0ee6-kube-api-access-jcpg9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6\" (UID: \"2d871e4c-674f-43fb-8c9e-5d57913f0ee6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.515011 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.514906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:25.635013 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.634976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:09:25.662867 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:09:25.662839 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d871e4c_674f_43fb_8c9e_5d57913f0ee6.slice/crio-ea2c89e93d4747ad519a37b86cf53832d4c825d18f20297925dad4835a50cf34 WatchSource:0}: Error finding container ea2c89e93d4747ad519a37b86cf53832d4c825d18f20297925dad4835a50cf34: Status 404 returned error can't find the container with id ea2c89e93d4747ad519a37b86cf53832d4c825d18f20297925dad4835a50cf34 Jul 19 15:09:25.664701 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:25.664554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6"] Jul 19 15:09:26.544898 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:26.544861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" event={"ID":"2d871e4c-674f-43fb-8c9e-5d57913f0ee6","Type":"ContainerStarted","Data":"b0de40f12cd9ddaadc30b36cb6ab4dd6ead25bcf1addd97b024b056570e562b8"} Jul 19 15:09:26.545281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:26.544912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" event={"ID":"2d871e4c-674f-43fb-8c9e-5d57913f0ee6","Type":"ContainerStarted","Data":"ea2c89e93d4747ad519a37b86cf53832d4c825d18f20297925dad4835a50cf34"} Jul 19 15:09:26.561915 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:26.561846 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" podStartSLOduration=1.373728539 podStartE2EDuration="1.561823789s" totalImagesPullingTime="188.09525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:09:26.560949489 +0000 UTC m=+717.425224289" watchObservedRunningTime="2026-07-19 15:09:26.561823789 +0000 UTC m=+717.426098587" Jul 19 15:09:35.515809 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:35.515770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:35.516212 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:35.515835 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:35.528299 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:35.528269 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:35.612541 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:35.612510 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6" Jul 19 15:09:38.401769 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.401727 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s"] Jul 19 15:09:38.406772 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.406747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.408805 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.408784 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 19 15:09:38.413136 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.413108 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s"] Jul 19 15:09:38.458189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.457801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.458189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.457846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.458189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.457873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.458189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.457952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6djq\" (UniqueName: \"kubernetes.io/projected/a18aabae-f4b9-4e39-a984-e9fd550685ed-kube-api-access-h6djq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.458189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.458054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.458189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.458148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18aabae-f4b9-4e39-a984-e9fd550685ed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.489806 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.489771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:09:38.559202 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559202 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6djq\" (UniqueName: \"kubernetes.io/projected/a18aabae-f4b9-4e39-a984-e9fd550685ed-kube-api-access-h6djq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18aabae-f4b9-4e39-a984-e9fd550685ed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559778 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559919 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.559996 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.559934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.561839 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.561815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18aabae-f4b9-4e39-a984-e9fd550685ed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.562011 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.561996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18aabae-f4b9-4e39-a984-e9fd550685ed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.569692 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.569664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6djq\" (UniqueName: \"kubernetes.io/projected/a18aabae-f4b9-4e39-a984-e9fd550685ed-kube-api-access-h6djq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qcc9s\" (UID: \"a18aabae-f4b9-4e39-a984-e9fd550685ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.719918 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.719828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:38.856843 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:38.856812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s"] Jul 19 15:09:38.858388 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:09:38.858357 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18aabae_f4b9_4e39_a984_e9fd550685ed.slice/crio-78cb6057210f44a4b59389392f42966891ca43e03c87431376f745c6310a201e WatchSource:0}: Error finding container 78cb6057210f44a4b59389392f42966891ca43e03c87431376f745c6310a201e: Status 404 returned error can't find the container with id 78cb6057210f44a4b59389392f42966891ca43e03c87431376f745c6310a201e Jul 19 15:09:39.626573 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:39.626535 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" event={"ID":"a18aabae-f4b9-4e39-a984-e9fd550685ed","Type":"ContainerStarted","Data":"c959d740a3b2686a7eb6e7202652728626292aa51da425fa0333359e38e165b8"} Jul 19 15:09:39.627016 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:39.626619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" event={"ID":"a18aabae-f4b9-4e39-a984-e9fd550685ed","Type":"ContainerStarted","Data":"78cb6057210f44a4b59389392f42966891ca43e03c87431376f745c6310a201e"} Jul 19 15:09:39.642779 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:39.642718 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" podStartSLOduration=1.461361574 podStartE2EDuration="1.642702766s" totalImagesPullingTime="181.341192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:09:39.64118778 +0000 UTC m=+730.505462577" watchObservedRunningTime="2026-07-19 15:09:39.642702766 +0000 UTC m=+730.506977562" Jul 19 15:09:42.537720 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:42.537683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:09:48.720152 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:48.720108 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:48.720636 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:48.720174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:48.733097 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:48.733073 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:09:49.698216 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:09:49.698189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qcc9s" Jul 19 15:10:43.259769 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:10:43.259735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:10:52.578680 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:10:52.578568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:10:52.674407 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:10:52.674362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:10:53.659260 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:10:53.659219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:10:53.757367 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:10:53.757329 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:11:40.016949 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.016905 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-j2fsc"] Jul 19 15:11:40.017527 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.017177 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" podUID="16efbd1a-a6c8-4110-a1bf-f914c672092d" containerName="manager" containerID="cri-o://c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0" gracePeriod=30 Jul 19 15:11:40.277327 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.277267 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:11:40.302032 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.301997 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqhg\" (UniqueName: \"kubernetes.io/projected/16efbd1a-a6c8-4110-a1bf-f914c672092d-kube-api-access-4nqhg\") pod \"16efbd1a-a6c8-4110-a1bf-f914c672092d\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " Jul 19 15:11:40.302217 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.302175 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16efbd1a-a6c8-4110-a1bf-f914c672092d-webhook-cert\") pod \"16efbd1a-a6c8-4110-a1bf-f914c672092d\" (UID: \"16efbd1a-a6c8-4110-a1bf-f914c672092d\") " Jul 19 15:11:40.304611 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.304555 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16efbd1a-a6c8-4110-a1bf-f914c672092d-kube-api-access-4nqhg" (OuterVolumeSpecName: "kube-api-access-4nqhg") pod "16efbd1a-a6c8-4110-a1bf-f914c672092d" (UID: "16efbd1a-a6c8-4110-a1bf-f914c672092d"). InnerVolumeSpecName "kube-api-access-4nqhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:11:40.304728 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.304609 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16efbd1a-a6c8-4110-a1bf-f914c672092d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "16efbd1a-a6c8-4110-a1bf-f914c672092d" (UID: "16efbd1a-a6c8-4110-a1bf-f914c672092d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:11:40.321345 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.321311 2580 generic.go:356] "Generic (PLEG): container finished" podID="16efbd1a-a6c8-4110-a1bf-f914c672092d" containerID="c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0" exitCode=0 Jul 19 15:11:40.321476 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.321357 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" Jul 19 15:11:40.321476 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.321357 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" event={"ID":"16efbd1a-a6c8-4110-a1bf-f914c672092d","Type":"ContainerDied","Data":"c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0"} Jul 19 15:11:40.321614 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.321497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-j2fsc" event={"ID":"16efbd1a-a6c8-4110-a1bf-f914c672092d","Type":"ContainerDied","Data":"4ebb7cd69b7aedb4f700b0c44d5d0f7891f6c2cebe9b505c783816f300420bab"} Jul 19 15:11:40.321614 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.321542 2580 scope.go:117] "RemoveContainer" containerID="c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0" Jul 19 15:11:40.333610 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.333575 2580 scope.go:117] "RemoveContainer" containerID="c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0" Jul 19 15:11:40.333945 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:11:40.333922 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0\": container with ID starting with c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0 not found: ID does not exist" containerID="c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0" Jul 19 15:11:40.334043 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.333954 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0"} err="failed to get container status \"c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0\": rpc error: code = NotFound desc = could not find container \"c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0\": container with ID starting with c1b24e43c12bac45ddaffda93ce9e6d32c40d47c8ff51c38f17dc6dddd14ebb0 not found: ID does not exist" Jul 19 15:11:40.344039 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.344015 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-j2fsc"] Jul 19 15:11:40.347998 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.347974 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-j2fsc"] Jul 19 15:11:40.403837 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.403807 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqhg\" (UniqueName: \"kubernetes.io/projected/16efbd1a-a6c8-4110-a1bf-f914c672092d-kube-api-access-4nqhg\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:11:40.403837 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:40.403836 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16efbd1a-a6c8-4110-a1bf-f914c672092d-webhook-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:11:41.662985 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:41.662951 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16efbd1a-a6c8-4110-a1bf-f914c672092d" path="/var/lib/kubelet/pods/16efbd1a-a6c8-4110-a1bf-f914c672092d/volumes" Jul 19 15:11:57.279112 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:11:57.279063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:12:04.906522 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:04.906485 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:12:05.007425 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:05.007387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:12:29.640363 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:29.640280 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:12:29.642048 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:29.642027 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:12:29.646711 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:29.646691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:12:29.648317 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:29.648299 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:12:59.610205 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:12:59.610173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:01.837640 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:01.837600 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:42.503682 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:42.503645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:42.600144 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:42.600108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:42.705259 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:42.705223 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:48.349608 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:48.349543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:48.452353 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:48.452321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:48.549116 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:48.549078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:48.695064 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:48.695010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:54.284081 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:54.284003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:13:54.346329 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:13:54.346293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:15:05.539478 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:15:05.539444 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:15:09.588399 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:15:09.588367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:15:09.637795 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:15:09.637764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:15:42.403897 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:15:42.403861 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:16:26.231538 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:16:26.231504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:17:29.678415 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:17:29.678385 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:17:29.684100 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:17:29.684077 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:17:29.684333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:17:29.684316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:17:29.689974 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:17:29.689955 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:17:39.762085 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:17:39.762041 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:20:28.503473 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:20:28.503393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:21:27.740414 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:27.740334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:21:40.413826 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.413789 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45"] Jul 19 15:21:40.414443 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.414427 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16efbd1a-a6c8-4110-a1bf-f914c672092d" containerName="manager" Jul 19 15:21:40.414487 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.414446 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="16efbd1a-a6c8-4110-a1bf-f914c672092d" containerName="manager" Jul 19 15:21:40.414570 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.414558 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="16efbd1a-a6c8-4110-a1bf-f914c672092d" containerName="manager" Jul 19 15:21:40.418069 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.418044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.420194 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.420173 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-95a34797-kserve-self-signed-certs\"" Jul 19 15:21:40.431868 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.431834 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45"] Jul 19 15:21:40.439130 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.439110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:21:40.572197 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.572153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-dshm\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.572398 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.572214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-tmp-dir\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.572398 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.572345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-home\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.572398 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.572378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-model-cache\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.572687 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.572414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ea682b-19ba-48c0-9bc3-4004186eba16-tls-certs\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.572687 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.572443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2z8j\" (UniqueName: \"kubernetes.io/projected/09ea682b-19ba-48c0-9bc3-4004186eba16-kube-api-access-x2z8j\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673272 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-home\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673272 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-model-cache\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673272 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ea682b-19ba-48c0-9bc3-4004186eba16-tls-certs\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673272 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2z8j\" (UniqueName: \"kubernetes.io/projected/09ea682b-19ba-48c0-9bc3-4004186eba16-kube-api-access-x2z8j\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673652 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-dshm\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673652 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-tmp-dir\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673771 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-home\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673771 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-model-cache\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.673841 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.673788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-tmp-dir\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.675617 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.675576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-dshm\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.675930 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.675909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ea682b-19ba-48c0-9bc3-4004186eba16-tls-certs\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.680773 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.680748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2z8j\" (UniqueName: \"kubernetes.io/projected/09ea682b-19ba-48c0-9bc3-4004186eba16-kube-api-access-x2z8j\") pod \"e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.726924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.726893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:40.867211 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:40.867182 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45"] Jul 19 15:21:40.868478 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:21:40.868457 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09ea682b_19ba_48c0_9bc3_4004186eba16.slice/crio-0690be0e791df142d30afd885fbbcdfffc224a9ea65cc7140359e8ba2a7766b6 WatchSource:0}: Error finding container 0690be0e791df142d30afd885fbbcdfffc224a9ea65cc7140359e8ba2a7766b6: Status 404 returned error can't find the container with id 0690be0e791df142d30afd885fbbcdfffc224a9ea65cc7140359e8ba2a7766b6 Jul 19 15:21:41.753492 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:41.753452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" event={"ID":"09ea682b-19ba-48c0-9bc3-4004186eba16","Type":"ContainerStarted","Data":"43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac"} Jul 19 15:21:41.753492 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:41.753496 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" event={"ID":"09ea682b-19ba-48c0-9bc3-4004186eba16","Type":"ContainerStarted","Data":"0690be0e791df142d30afd885fbbcdfffc224a9ea65cc7140359e8ba2a7766b6"} Jul 19 15:21:41.770079 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:41.770032 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" podStartSLOduration=1.770016282 podStartE2EDuration="1.770016282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:21:41.768364814 +0000 UTC m=+1452.632639629" watchObservedRunningTime="2026-07-19 15:21:41.770016282 +0000 UTC m=+1452.634291078" Jul 19 15:21:50.727696 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:50.727598 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:50.727696 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:50.727664 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:50.740324 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:50.740300 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:50.818619 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:50.818567 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:21:52.030038 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:21:52.029975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:22:29.717919 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:29.717892 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:22:29.723711 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:29.723690 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:22:29.724894 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:29.724878 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:22:29.730847 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:29.730832 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:22:36.293373 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:36.293336 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45"] Jul 19 15:22:37.097498 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:37.097462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:22:37.223424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:37.223393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:22:51.314498 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.314447 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" podUID="09ea682b-19ba-48c0-9bc3-4004186eba16" containerName="main" containerID="cri-o://43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac" gracePeriod=45 Jul 19 15:22:51.573261 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.573208 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:22:51.624879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.624848 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-tmp-dir\") pod \"09ea682b-19ba-48c0-9bc3-4004186eba16\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " Jul 19 15:22:51.624879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.624882 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2z8j\" (UniqueName: \"kubernetes.io/projected/09ea682b-19ba-48c0-9bc3-4004186eba16-kube-api-access-x2z8j\") pod \"09ea682b-19ba-48c0-9bc3-4004186eba16\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " Jul 19 15:22:51.625118 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.624904 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-model-cache\") pod \"09ea682b-19ba-48c0-9bc3-4004186eba16\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " Jul 19 15:22:51.625118 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.624957 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ea682b-19ba-48c0-9bc3-4004186eba16-tls-certs\") pod \"09ea682b-19ba-48c0-9bc3-4004186eba16\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " Jul 19 15:22:51.625118 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.624975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-home\") pod \"09ea682b-19ba-48c0-9bc3-4004186eba16\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " Jul 19 15:22:51.625118 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.624993 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-dshm\") pod \"09ea682b-19ba-48c0-9bc3-4004186eba16\" (UID: \"09ea682b-19ba-48c0-9bc3-4004186eba16\") " Jul 19 15:22:51.625333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.625178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09ea682b-19ba-48c0-9bc3-4004186eba16" (UID: "09ea682b-19ba-48c0-9bc3-4004186eba16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:51.625333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.625227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-home" (OuterVolumeSpecName: "home") pod "09ea682b-19ba-48c0-9bc3-4004186eba16" (UID: "09ea682b-19ba-48c0-9bc3-4004186eba16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:51.625333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.625270 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-model-cache" (OuterVolumeSpecName: "model-cache") pod "09ea682b-19ba-48c0-9bc3-4004186eba16" (UID: "09ea682b-19ba-48c0-9bc3-4004186eba16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:51.625333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.625325 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-tmp-dir\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:22:51.625527 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.625346 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-home\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:22:51.627221 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.627191 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ea682b-19ba-48c0-9bc3-4004186eba16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09ea682b-19ba-48c0-9bc3-4004186eba16" (UID: "09ea682b-19ba-48c0-9bc3-4004186eba16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:22:51.627336 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.627245 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-dshm" (OuterVolumeSpecName: "dshm") pod "09ea682b-19ba-48c0-9bc3-4004186eba16" (UID: "09ea682b-19ba-48c0-9bc3-4004186eba16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:51.627336 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.627262 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ea682b-19ba-48c0-9bc3-4004186eba16-kube-api-access-x2z8j" (OuterVolumeSpecName: "kube-api-access-x2z8j") pod "09ea682b-19ba-48c0-9bc3-4004186eba16" (UID: "09ea682b-19ba-48c0-9bc3-4004186eba16"). InnerVolumeSpecName "kube-api-access-x2z8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:22:51.726095 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.726065 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ea682b-19ba-48c0-9bc3-4004186eba16-tls-certs\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:22:51.726095 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.726087 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-dshm\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:22:51.726095 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.726096 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2z8j\" (UniqueName: \"kubernetes.io/projected/09ea682b-19ba-48c0-9bc3-4004186eba16-kube-api-access-x2z8j\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:22:51.726095 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:51.726105 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ea682b-19ba-48c0-9bc3-4004186eba16-model-cache\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:22:52.164335 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.164258 2580 generic.go:356] "Generic (PLEG): container finished" podID="09ea682b-19ba-48c0-9bc3-4004186eba16" containerID="43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac" exitCode=0 Jul 19 15:22:52.164335 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.164306 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" Jul 19 15:22:52.164535 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.164345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" event={"ID":"09ea682b-19ba-48c0-9bc3-4004186eba16","Type":"ContainerDied","Data":"43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac"} Jul 19 15:22:52.164535 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.164393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45" event={"ID":"09ea682b-19ba-48c0-9bc3-4004186eba16","Type":"ContainerDied","Data":"0690be0e791df142d30afd885fbbcdfffc224a9ea65cc7140359e8ba2a7766b6"} Jul 19 15:22:52.164535 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.164413 2580 scope.go:117] "RemoveContainer" containerID="43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac" Jul 19 15:22:52.174161 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.174143 2580 scope.go:117] "RemoveContainer" containerID="43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac" Jul 19 15:22:52.174423 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:22:52.174404 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac\": container with ID starting with 43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac not found: ID does not exist" containerID="43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac" Jul 19 15:22:52.174472 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.174433 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac"} err="failed to get container status \"43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac\": rpc error: code = NotFound desc = could not find container \"43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac\": container with ID starting with 43dea189fca191d63ddada796afb609b54de64a1fddd51bf8d36104578d6eeac not found: ID does not exist" Jul 19 15:22:52.179909 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.179887 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45"] Jul 19 15:22:52.182361 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:52.182344 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-95a34797-kserve-94d9589ff-8nx45"] Jul 19 15:22:53.663052 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:22:53.663009 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ea682b-19ba-48c0-9bc3-4004186eba16" path="/var/lib/kubelet/pods/09ea682b-19ba-48c0-9bc3-4004186eba16/volumes" Jul 19 15:25:03.724614 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:25:03.724528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:25:57.739294 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:25:57.739197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:07.673028 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:07.672995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:15.928513 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:15.928478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:21.036785 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:21.036741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:24.006596 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:24.006535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:24.104459 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:24.104424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:24.444334 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:24.444297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:29.958770 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:29.958716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:30.018783 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:30.018748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:30.185736 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:30.185696 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:30.503663 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:30.503626 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:30.603732 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:30.603700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:35.798395 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:35.798363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:35.902540 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:35.902509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.004446 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.004413 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg"] Jul 19 15:26:36.005358 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.005331 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ea682b-19ba-48c0-9bc3-4004186eba16" containerName="main" Jul 19 15:26:36.005509 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.005496 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ea682b-19ba-48c0-9bc3-4004186eba16" containerName="main" Jul 19 15:26:36.005972 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.005954 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ea682b-19ba-48c0-9bc3-4004186eba16" containerName="main" Jul 19 15:26:36.017050 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.017018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.019509 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.019482 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-44de5c81-serving-cert\"" Jul 19 15:26:36.019704 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.019610 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg"] Jul 19 15:26:36.036772 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.036748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.103619 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.103504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ff9r\" (UniqueName: \"kubernetes.io/projected/39341067-2027-4c18-aaec-a27a56f6d820-kube-api-access-9ff9r\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.103787 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.103613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.103787 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.103664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39341067-2027-4c18-aaec-a27a56f6d820-openshift-service-ca\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.205060 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.205023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff9r\" (UniqueName: \"kubernetes.io/projected/39341067-2027-4c18-aaec-a27a56f6d820-kube-api-access-9ff9r\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.205266 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.205064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.205266 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.205111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39341067-2027-4c18-aaec-a27a56f6d820-openshift-service-ca\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.205266 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:26:36.205249 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-serving-cert: secret "maas-api-e2e-ait-44de5c81-serving-cert" not found Jul 19 15:26:36.205449 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:26:36.205325 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls podName:39341067-2027-4c18-aaec-a27a56f6d820 nodeName:}" failed. No retries permitted until 2026-07-19 15:26:36.705307486 +0000 UTC m=+1747.569582261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls") pod "maas-api-e2e-ait-44de5c81-567764744-69rlg" (UID: "39341067-2027-4c18-aaec-a27a56f6d820") : secret "maas-api-e2e-ait-44de5c81-serving-cert" not found Jul 19 15:26:36.205851 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.205828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39341067-2027-4c18-aaec-a27a56f6d820-openshift-service-ca\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.214740 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.214713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ff9r\" (UniqueName: \"kubernetes.io/projected/39341067-2027-4c18-aaec-a27a56f6d820-kube-api-access-9ff9r\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.447025 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.446990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.544879 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.544852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.646609 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.646569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.709607 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.709489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.712072 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.712052 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls\") pod \"maas-api-e2e-ait-44de5c81-567764744-69rlg\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:36.824402 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.824364 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.926492 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.926462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:26:36.928504 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:36.928487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:37.068709 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:37.068684 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg"] Jul 19 15:26:37.069767 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:26:37.069738 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39341067_2027_4c18_aaec_a27a56f6d820.slice/crio-be0af5802ae62fce68994379af68800bdaccc6f0528ba94e0bccd0492f9a7b79 WatchSource:0}: Error finding container be0af5802ae62fce68994379af68800bdaccc6f0528ba94e0bccd0492f9a7b79: Status 404 returned error can't find the container with id be0af5802ae62fce68994379af68800bdaccc6f0528ba94e0bccd0492f9a7b79 Jul 19 15:26:37.071050 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:37.071030 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:26:37.467393 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:37.467359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" event={"ID":"39341067-2027-4c18-aaec-a27a56f6d820","Type":"ContainerStarted","Data":"be0af5802ae62fce68994379af68800bdaccc6f0528ba94e0bccd0492f9a7b79"} Jul 19 15:26:38.474422 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:38.474384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" event={"ID":"39341067-2027-4c18-aaec-a27a56f6d820","Type":"ContainerStarted","Data":"bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7"} Jul 19 15:26:38.474895 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:38.474471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:26:38.492742 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:38.492686 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" podStartSLOduration=3.094077861 podStartE2EDuration="3.492668712s" totalImagesPullingTime="398.590851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:26:38.490233376 +0000 UTC m=+1749.354508172" watchObservedRunningTime="2026-07-19 15:26:38.492668712 +0000 UTC m=+1749.356943509" Jul 19 15:26:44.485308 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:26:44.485277 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:27:11.648425 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:11.648388 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg"] Jul 19 15:27:11.649014 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:11.648738 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" podUID="39341067-2027-4c18-aaec-a27a56f6d820" containerName="maas-api" containerID="cri-o://bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7" gracePeriod=30 Jul 19 15:27:11.666923 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:11.666896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:11.908512 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:11.908446 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:27:12.037267 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.037213 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls\") pod \"39341067-2027-4c18-aaec-a27a56f6d820\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " Jul 19 15:27:12.037475 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.037371 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ff9r\" (UniqueName: \"kubernetes.io/projected/39341067-2027-4c18-aaec-a27a56f6d820-kube-api-access-9ff9r\") pod \"39341067-2027-4c18-aaec-a27a56f6d820\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " Jul 19 15:27:12.037475 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.037399 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39341067-2027-4c18-aaec-a27a56f6d820-openshift-service-ca\") pod \"39341067-2027-4c18-aaec-a27a56f6d820\" (UID: \"39341067-2027-4c18-aaec-a27a56f6d820\") " Jul 19 15:27:12.037842 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.037817 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39341067-2027-4c18-aaec-a27a56f6d820-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39341067-2027-4c18-aaec-a27a56f6d820" (UID: "39341067-2027-4c18-aaec-a27a56f6d820"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:27:12.039507 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.039483 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39341067-2027-4c18-aaec-a27a56f6d820" (UID: "39341067-2027-4c18-aaec-a27a56f6d820"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:27:12.039507 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.039497 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39341067-2027-4c18-aaec-a27a56f6d820-kube-api-access-9ff9r" (OuterVolumeSpecName: "kube-api-access-9ff9r") pod "39341067-2027-4c18-aaec-a27a56f6d820" (UID: "39341067-2027-4c18-aaec-a27a56f6d820"). InnerVolumeSpecName "kube-api-access-9ff9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:27:12.138550 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.138514 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ff9r\" (UniqueName: \"kubernetes.io/projected/39341067-2027-4c18-aaec-a27a56f6d820-kube-api-access-9ff9r\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:27:12.138550 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.138541 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39341067-2027-4c18-aaec-a27a56f6d820-openshift-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:27:12.138550 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.138552 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39341067-2027-4c18-aaec-a27a56f6d820-maas-api-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:27:12.678087 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.678053 2580 generic.go:356] "Generic (PLEG): container finished" podID="39341067-2027-4c18-aaec-a27a56f6d820" containerID="bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7" exitCode=0 Jul 19 15:27:12.678546 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.678096 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" Jul 19 15:27:12.678546 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.678144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" event={"ID":"39341067-2027-4c18-aaec-a27a56f6d820","Type":"ContainerDied","Data":"bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7"} Jul 19 15:27:12.678546 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.678208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg" event={"ID":"39341067-2027-4c18-aaec-a27a56f6d820","Type":"ContainerDied","Data":"be0af5802ae62fce68994379af68800bdaccc6f0528ba94e0bccd0492f9a7b79"} Jul 19 15:27:12.678546 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.678246 2580 scope.go:117] "RemoveContainer" containerID="bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7" Jul 19 15:27:12.688315 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.688295 2580 scope.go:117] "RemoveContainer" containerID="bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7" Jul 19 15:27:12.688576 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:27:12.688557 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7\": container with ID starting with bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7 not found: ID does not exist" containerID="bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7" Jul 19 15:27:12.688664 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.688615 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7"} err="failed to get container status \"bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7\": rpc error: code = NotFound desc = could not find container \"bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7\": container with ID starting with bf52696387dd860fcad67244268c67f32b45dd04be87d00a6652739d8373f7f7 not found: ID does not exist" Jul 19 15:27:12.700335 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.700315 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg"] Jul 19 15:27:12.703869 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:12.703848 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-44de5c81-567764744-69rlg"] Jul 19 15:27:13.663185 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:13.663149 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39341067-2027-4c18-aaec-a27a56f6d820" path="/var/lib/kubelet/pods/39341067-2027-4c18-aaec-a27a56f6d820/volumes" Jul 19 15:27:16.359520 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:16.359479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:16.502363 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:16.502326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:16.736609 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:16.736504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:17.036101 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:17.036023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:17.173597 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:17.173559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:17.280875 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:17.280840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:17.540221 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:17.540188 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:18.002033 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:18.002004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:20.004713 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.004683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:20.082597 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.082544 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc"] Jul 19 15:27:20.083215 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.083191 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39341067-2027-4c18-aaec-a27a56f6d820" containerName="maas-api" Jul 19 15:27:20.083215 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.083215 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39341067-2027-4c18-aaec-a27a56f6d820" containerName="maas-api" Jul 19 15:27:20.083387 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.083326 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39341067-2027-4c18-aaec-a27a56f6d820" containerName="maas-api" Jul 19 15:27:20.087920 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.087898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.090140 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.090121 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-431757e3-serving-cert\"" Jul 19 15:27:20.097360 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.097339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc"] Jul 19 15:27:20.113486 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.113462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95598efa-5002-47d7-b668-83980b8469bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.113615 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.113497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64sx\" (UniqueName: \"kubernetes.io/projected/95598efa-5002-47d7-b668-83980b8469bc-kube-api-access-v64sx\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.113615 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.113541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95598efa-5002-47d7-b668-83980b8469bc-maas-api-tls\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.115378 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.115357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:20.214773 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.214740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64sx\" (UniqueName: \"kubernetes.io/projected/95598efa-5002-47d7-b668-83980b8469bc-kube-api-access-v64sx\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.214968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.214809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95598efa-5002-47d7-b668-83980b8469bc-maas-api-tls\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.214968 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.214908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95598efa-5002-47d7-b668-83980b8469bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.215488 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.215470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95598efa-5002-47d7-b668-83980b8469bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.217278 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.217249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95598efa-5002-47d7-b668-83980b8469bc-maas-api-tls\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.224905 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.224881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64sx\" (UniqueName: \"kubernetes.io/projected/95598efa-5002-47d7-b668-83980b8469bc-kube-api-access-v64sx\") pod \"maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.255792 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.255702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:20.400463 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.400432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:20.548696 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.548672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc"] Jul 19 15:27:20.549947 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:27:20.549920 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95598efa_5002_47d7_b668_83980b8469bc.slice/crio-5e710399ef25d9b81cf5f64e1e2a46d1d7b6e9ec518fac087aef7665f67bd99c WatchSource:0}: Error finding container 5e710399ef25d9b81cf5f64e1e2a46d1d7b6e9ec518fac087aef7665f67bd99c: Status 404 returned error can't find the container with id 5e710399ef25d9b81cf5f64e1e2a46d1d7b6e9ec518fac087aef7665f67bd99c Jul 19 15:27:20.593332 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.593300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:20.694780 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.694751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:20.736092 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:20.736055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" event={"ID":"95598efa-5002-47d7-b668-83980b8469bc","Type":"ContainerStarted","Data":"5e710399ef25d9b81cf5f64e1e2a46d1d7b6e9ec518fac087aef7665f67bd99c"} Jul 19 15:27:21.743821 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:21.743759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" event={"ID":"95598efa-5002-47d7-b668-83980b8469bc","Type":"ContainerStarted","Data":"e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13"} Jul 19 15:27:21.744228 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:21.743899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:21.762564 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:21.762519 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" podStartSLOduration=1.400696928 podStartE2EDuration="1.762502829s" totalImagesPullingTime="361.805901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:27:21.759642672 +0000 UTC m=+1792.623917468" watchObservedRunningTime="2026-07-19 15:27:21.762502829 +0000 UTC m=+1792.626777625" Jul 19 15:27:27.754932 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:27.754901 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:29.757647 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:29.757619 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:27:29.764189 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:29.764167 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:27:29.766496 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:29.766478 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:27:29.772371 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:29.772354 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:27:55.427778 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.427744 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc"] Jul 19 15:27:55.428360 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.428000 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" podUID="95598efa-5002-47d7-b668-83980b8469bc" containerName="maas-api" containerID="cri-o://e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13" gracePeriod=30 Jul 19 15:27:55.447885 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.447834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:27:55.683758 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.683703 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:55.847933 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.847896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95598efa-5002-47d7-b668-83980b8469bc-openshift-service-ca\") pod \"95598efa-5002-47d7-b668-83980b8469bc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " Jul 19 15:27:55.848108 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.847960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95598efa-5002-47d7-b668-83980b8469bc-maas-api-tls\") pod \"95598efa-5002-47d7-b668-83980b8469bc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " Jul 19 15:27:55.848108 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.847998 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v64sx\" (UniqueName: \"kubernetes.io/projected/95598efa-5002-47d7-b668-83980b8469bc-kube-api-access-v64sx\") pod \"95598efa-5002-47d7-b668-83980b8469bc\" (UID: \"95598efa-5002-47d7-b668-83980b8469bc\") " Jul 19 15:27:55.848329 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.848303 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95598efa-5002-47d7-b668-83980b8469bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95598efa-5002-47d7-b668-83980b8469bc" (UID: "95598efa-5002-47d7-b668-83980b8469bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:27:55.850182 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.850159 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95598efa-5002-47d7-b668-83980b8469bc-kube-api-access-v64sx" (OuterVolumeSpecName: "kube-api-access-v64sx") pod "95598efa-5002-47d7-b668-83980b8469bc" (UID: "95598efa-5002-47d7-b668-83980b8469bc"). InnerVolumeSpecName "kube-api-access-v64sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:27:55.850269 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.850196 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95598efa-5002-47d7-b668-83980b8469bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95598efa-5002-47d7-b668-83980b8469bc" (UID: "95598efa-5002-47d7-b668-83980b8469bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:27:55.940776 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.940679 2580 generic.go:356] "Generic (PLEG): container finished" podID="95598efa-5002-47d7-b668-83980b8469bc" containerID="e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13" exitCode=0 Jul 19 15:27:55.940776 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.940726 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" Jul 19 15:27:55.940776 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.940760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" event={"ID":"95598efa-5002-47d7-b668-83980b8469bc","Type":"ContainerDied","Data":"e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13"} Jul 19 15:27:55.941071 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.940816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc" event={"ID":"95598efa-5002-47d7-b668-83980b8469bc","Type":"ContainerDied","Data":"5e710399ef25d9b81cf5f64e1e2a46d1d7b6e9ec518fac087aef7665f67bd99c"} Jul 19 15:27:55.941071 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.940841 2580 scope.go:117] "RemoveContainer" containerID="e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13" Jul 19 15:27:55.948842 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.948820 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95598efa-5002-47d7-b668-83980b8469bc-openshift-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:27:55.948842 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.948842 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95598efa-5002-47d7-b668-83980b8469bc-maas-api-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:27:55.948991 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.948852 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64sx\" (UniqueName: \"kubernetes.io/projected/95598efa-5002-47d7-b668-83980b8469bc-kube-api-access-v64sx\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:27:55.951168 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.951153 2580 scope.go:117] "RemoveContainer" containerID="e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13" Jul 19 15:27:55.951405 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:27:55.951388 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13\": container with ID starting with e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13 not found: ID does not exist" containerID="e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13" Jul 19 15:27:55.951457 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.951414 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13"} err="failed to get container status \"e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13\": rpc error: code = NotFound desc = could not find container \"e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13\": container with ID starting with e54401a9d912b99a022733e3b8f0df4a526e47095cc79ac963b34f4f6fdedd13 not found: ID does not exist" Jul 19 15:27:55.962608 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.962565 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc"] Jul 19 15:27:55.968455 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:55.968435 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-431757e3-5fb6f99f7d-bjvkc"] Jul 19 15:27:57.663287 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:27:57.663253 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95598efa-5002-47d7-b668-83980b8469bc" path="/var/lib/kubelet/pods/95598efa-5002-47d7-b668-83980b8469bc/volumes" Jul 19 15:28:00.104699 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:00.104653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:00.261662 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:00.261633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:00.504553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:00.504523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:00.742325 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:00.742289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:01.082380 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:01.082349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:01.235225 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:01.235190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:03.079564 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.079526 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:03.152065 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.152032 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw"] Jul 19 15:28:03.152594 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.152556 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95598efa-5002-47d7-b668-83980b8469bc" containerName="maas-api" Jul 19 15:28:03.152731 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.152599 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95598efa-5002-47d7-b668-83980b8469bc" containerName="maas-api" Jul 19 15:28:03.152791 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.152728 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95598efa-5002-47d7-b668-83980b8469bc" containerName="maas-api" Jul 19 15:28:03.155794 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.155746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:03.155794 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.155788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.158094 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.158071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a4b8edd0-serving-cert\"" Jul 19 15:28:03.168159 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.168136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw"] Jul 19 15:28:03.210416 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.210378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8b7871-4f91-403b-8963-951c6bfaccfe-openshift-service-ca\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.210630 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.210453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.210630 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.210473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbfn\" (UniqueName: \"kubernetes.io/projected/4b8b7871-4f91-403b-8963-951c6bfaccfe-kube-api-access-lcbfn\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.240378 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.240349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:03.311464 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.311426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.311464 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.311466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbfn\" (UniqueName: \"kubernetes.io/projected/4b8b7871-4f91-403b-8963-951c6bfaccfe-kube-api-access-lcbfn\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.311707 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.311520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8b7871-4f91-403b-8963-951c6bfaccfe-openshift-service-ca\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.311707 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:28:03.311607 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-serving-cert: secret "maas-api-e2e-ait-a4b8edd0-serving-cert" not found Jul 19 15:28:03.311707 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:28:03.311678 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls podName:4b8b7871-4f91-403b-8963-951c6bfaccfe nodeName:}" failed. No retries permitted until 2026-07-19 15:28:03.811662617 +0000 UTC m=+1834.675937396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls") pod "maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" (UID: "4b8b7871-4f91-403b-8963-951c6bfaccfe") : secret "maas-api-e2e-ait-a4b8edd0-serving-cert" not found Jul 19 15:28:03.312131 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.312114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8b7871-4f91-403b-8963-951c6bfaccfe-openshift-service-ca\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.319534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.319510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbfn\" (UniqueName: \"kubernetes.io/projected/4b8b7871-4f91-403b-8963-951c6bfaccfe-kube-api-access-lcbfn\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.397389 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.397356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:03.815389 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.815298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.817815 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.817796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls\") pod \"maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:03.908963 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.908927 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:03.997213 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:03.997179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:04.068530 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:04.068442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:04.209036 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:04.209007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw"] Jul 19 15:28:04.210147 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:28:04.210115 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8b7871_4f91_403b_8963_951c6bfaccfe.slice/crio-2d87e78e445d92640bbf68a373ed556c940ed0d7d190645f5a4dbb88fcb3323b WatchSource:0}: Error finding container 2d87e78e445d92640bbf68a373ed556c940ed0d7d190645f5a4dbb88fcb3323b: Status 404 returned error can't find the container with id 2d87e78e445d92640bbf68a373ed556c940ed0d7d190645f5a4dbb88fcb3323b Jul 19 15:28:04.993343 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:04.993261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" event={"ID":"4b8b7871-4f91-403b-8963-951c6bfaccfe","Type":"ContainerStarted","Data":"60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86"} Jul 19 15:28:04.993343 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:04.993319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" event={"ID":"4b8b7871-4f91-403b-8963-951c6bfaccfe","Type":"ContainerStarted","Data":"2d87e78e445d92640bbf68a373ed556c940ed0d7d190645f5a4dbb88fcb3323b"} Jul 19 15:28:04.993533 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:04.993351 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:05.010958 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:05.010907 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" podStartSLOduration=1.569453322 podStartE2EDuration="2.01089187s" totalImagesPullingTime="441.438548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:28:05.00813083 +0000 UTC m=+1835.872405626" watchObservedRunningTime="2026-07-19 15:28:05.01089187 +0000 UTC m=+1835.875166666" Jul 19 15:28:11.004645 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:11.004613 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:23.859497 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:23.859462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:23.960296 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:23.960262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:24.673202 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:24.673165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:24.774787 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:24.774749 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:39.412429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.412381 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw"] Jul 19 15:28:39.413302 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.413266 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" podUID="4b8b7871-4f91-403b-8963-951c6bfaccfe" containerName="maas-api" containerID="cri-o://60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86" gracePeriod=30 Jul 19 15:28:39.440158 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.440130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:39.679038 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.679017 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:39.842181 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.842139 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls\") pod \"4b8b7871-4f91-403b-8963-951c6bfaccfe\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " Jul 19 15:28:39.842384 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.842197 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8b7871-4f91-403b-8963-951c6bfaccfe-openshift-service-ca\") pod \"4b8b7871-4f91-403b-8963-951c6bfaccfe\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " Jul 19 15:28:39.842384 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.842240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcbfn\" (UniqueName: \"kubernetes.io/projected/4b8b7871-4f91-403b-8963-951c6bfaccfe-kube-api-access-lcbfn\") pod \"4b8b7871-4f91-403b-8963-951c6bfaccfe\" (UID: \"4b8b7871-4f91-403b-8963-951c6bfaccfe\") " Jul 19 15:28:39.842572 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.842541 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8b7871-4f91-403b-8963-951c6bfaccfe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b8b7871-4f91-403b-8963-951c6bfaccfe" (UID: "4b8b7871-4f91-403b-8963-951c6bfaccfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:28:39.844458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.844422 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b8b7871-4f91-403b-8963-951c6bfaccfe" (UID: "4b8b7871-4f91-403b-8963-951c6bfaccfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:28:39.844458 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.844426 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8b7871-4f91-403b-8963-951c6bfaccfe-kube-api-access-lcbfn" (OuterVolumeSpecName: "kube-api-access-lcbfn") pod "4b8b7871-4f91-403b-8963-951c6bfaccfe" (UID: "4b8b7871-4f91-403b-8963-951c6bfaccfe"). InnerVolumeSpecName "kube-api-access-lcbfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:28:39.943566 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.943475 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8b7871-4f91-403b-8963-951c6bfaccfe-maas-api-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:28:39.943566 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.943503 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8b7871-4f91-403b-8963-951c6bfaccfe-openshift-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:28:39.943566 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:39.943513 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbfn\" (UniqueName: \"kubernetes.io/projected/4b8b7871-4f91-403b-8963-951c6bfaccfe-kube-api-access-lcbfn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:28:40.208355 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.208269 2580 generic.go:356] "Generic (PLEG): container finished" podID="4b8b7871-4f91-403b-8963-951c6bfaccfe" containerID="60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86" exitCode=0 Jul 19 15:28:40.208355 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.208314 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" Jul 19 15:28:40.208533 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.208365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" event={"ID":"4b8b7871-4f91-403b-8963-951c6bfaccfe","Type":"ContainerDied","Data":"60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86"} Jul 19 15:28:40.208533 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.208417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw" event={"ID":"4b8b7871-4f91-403b-8963-951c6bfaccfe","Type":"ContainerDied","Data":"2d87e78e445d92640bbf68a373ed556c940ed0d7d190645f5a4dbb88fcb3323b"} Jul 19 15:28:40.208533 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.208441 2580 scope.go:117] "RemoveContainer" containerID="60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86" Jul 19 15:28:40.219026 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.219009 2580 scope.go:117] "RemoveContainer" containerID="60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86" Jul 19 15:28:40.220040 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:28:40.220015 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86\": container with ID starting with 60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86 not found: ID does not exist" containerID="60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86" Jul 19 15:28:40.220151 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.220046 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86"} err="failed to get container status \"60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86\": rpc error: code = NotFound desc = could not find container \"60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86\": container with ID starting with 60697acd2373879a71491a73470776395844be99b681b1f877962c30c0d00c86 not found: ID does not exist" Jul 19 15:28:40.231687 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.231667 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw"] Jul 19 15:28:40.235656 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:40.235633 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a4b8edd0-5bcc7c5c8-v5ghw"] Jul 19 15:28:41.663229 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:41.663198 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8b7871-4f91-403b-8963-951c6bfaccfe" path="/var/lib/kubelet/pods/4b8b7871-4f91-403b-8963-951c6bfaccfe/volumes" Jul 19 15:28:43.957477 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:43.957438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:45.175329 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:45.175287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:45.397823 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:45.397790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:45.619699 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:45.619674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:45.927738 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:45.927606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:46.160851 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:46.160810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:48.287120 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.287084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:48.355262 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.355233 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88"] Jul 19 15:28:48.355711 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.355695 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8b7871-4f91-403b-8963-951c6bfaccfe" containerName="maas-api" Jul 19 15:28:48.355711 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.355712 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8b7871-4f91-403b-8963-951c6bfaccfe" containerName="maas-api" Jul 19 15:28:48.355850 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.355788 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8b7871-4f91-403b-8963-951c6bfaccfe" containerName="maas-api" Jul 19 15:28:48.360304 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.360286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.362622 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.362569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:48.362891 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.362874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e7bec00b-serving-cert\"" Jul 19 15:28:48.371850 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.371826 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88"] Jul 19 15:28:48.437778 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.437744 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:48.522126 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.522087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cn6\" (UniqueName: \"kubernetes.io/projected/c82e4ac3-b83f-478d-94a6-104868225497-kube-api-access-n2cn6\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.522281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.522132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c82e4ac3-b83f-478d-94a6-104868225497-maas-api-tls\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.522281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.522249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82e4ac3-b83f-478d-94a6-104868225497-openshift-service-ca\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.544440 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.544350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:48.623750 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.623711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82e4ac3-b83f-478d-94a6-104868225497-openshift-service-ca\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.623953 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.623796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cn6\" (UniqueName: \"kubernetes.io/projected/c82e4ac3-b83f-478d-94a6-104868225497-kube-api-access-n2cn6\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.623953 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.623824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c82e4ac3-b83f-478d-94a6-104868225497-maas-api-tls\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.624396 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.624372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82e4ac3-b83f-478d-94a6-104868225497-openshift-service-ca\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.626320 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.626297 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c82e4ac3-b83f-478d-94a6-104868225497-maas-api-tls\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.632702 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.632682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cn6\" (UniqueName: \"kubernetes.io/projected/c82e4ac3-b83f-478d-94a6-104868225497-kube-api-access-n2cn6\") pod \"maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.671910 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.671874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:48.824273 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.824177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88"] Jul 19 15:28:48.830868 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:28:48.830841 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82e4ac3_b83f_478d_94a6_104868225497.slice/crio-9beffcdcc69aeeebb681393f69889b57a83e0bda5fe8bf87991e044ce57c9fd5 WatchSource:0}: Error finding container 9beffcdcc69aeeebb681393f69889b57a83e0bda5fe8bf87991e044ce57c9fd5: Status 404 returned error can't find the container with id 9beffcdcc69aeeebb681393f69889b57a83e0bda5fe8bf87991e044ce57c9fd5 Jul 19 15:28:48.931734 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:48.931700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:49.030899 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:49.030868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:28:49.263641 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:49.263603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" event={"ID":"c82e4ac3-b83f-478d-94a6-104868225497","Type":"ContainerStarted","Data":"9beffcdcc69aeeebb681393f69889b57a83e0bda5fe8bf87991e044ce57c9fd5"} Jul 19 15:28:50.272104 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:50.272074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" event={"ID":"c82e4ac3-b83f-478d-94a6-104868225497","Type":"ContainerStarted","Data":"7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58"} Jul 19 15:28:50.272464 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:50.272146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:28:50.289779 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:50.289726 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" podStartSLOduration=1.7824132129999999 podStartE2EDuration="2.289709803s" totalImagesPullingTime="507.29659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:28:50.286461513 +0000 UTC m=+1881.150736311" watchObservedRunningTime="2026-07-19 15:28:50.289709803 +0000 UTC m=+1881.153984600" Jul 19 15:28:56.284419 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:28:56.284344 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:29:23.508559 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.508519 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88"] Jul 19 15:29:23.509951 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.509883 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" podUID="c82e4ac3-b83f-478d-94a6-104868225497" containerName="maas-api" containerID="cri-o://7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58" gracePeriod=30 Jul 19 15:29:23.532747 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.532724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:23.767015 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.766955 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:29:23.857331 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.857291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2cn6\" (UniqueName: \"kubernetes.io/projected/c82e4ac3-b83f-478d-94a6-104868225497-kube-api-access-n2cn6\") pod \"c82e4ac3-b83f-478d-94a6-104868225497\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " Jul 19 15:29:23.857481 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.857415 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82e4ac3-b83f-478d-94a6-104868225497-openshift-service-ca\") pod \"c82e4ac3-b83f-478d-94a6-104868225497\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " Jul 19 15:29:23.857481 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.857451 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c82e4ac3-b83f-478d-94a6-104868225497-maas-api-tls\") pod \"c82e4ac3-b83f-478d-94a6-104868225497\" (UID: \"c82e4ac3-b83f-478d-94a6-104868225497\") " Jul 19 15:29:23.857810 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.857788 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82e4ac3-b83f-478d-94a6-104868225497-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c82e4ac3-b83f-478d-94a6-104868225497" (UID: "c82e4ac3-b83f-478d-94a6-104868225497"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:29:23.859563 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.859542 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82e4ac3-b83f-478d-94a6-104868225497-kube-api-access-n2cn6" (OuterVolumeSpecName: "kube-api-access-n2cn6") pod "c82e4ac3-b83f-478d-94a6-104868225497" (UID: "c82e4ac3-b83f-478d-94a6-104868225497"). InnerVolumeSpecName "kube-api-access-n2cn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:29:23.859672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.859565 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82e4ac3-b83f-478d-94a6-104868225497-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c82e4ac3-b83f-478d-94a6-104868225497" (UID: "c82e4ac3-b83f-478d-94a6-104868225497"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:29:23.958440 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.958408 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82e4ac3-b83f-478d-94a6-104868225497-openshift-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:29:23.958440 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.958435 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c82e4ac3-b83f-478d-94a6-104868225497-maas-api-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:29:23.958440 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:23.958444 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2cn6\" (UniqueName: \"kubernetes.io/projected/c82e4ac3-b83f-478d-94a6-104868225497-kube-api-access-n2cn6\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:29:24.469082 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.469043 2580 generic.go:356] "Generic (PLEG): container finished" podID="c82e4ac3-b83f-478d-94a6-104868225497" containerID="7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58" exitCode=0 Jul 19 15:29:24.469286 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.469087 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" Jul 19 15:29:24.469286 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.469126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" event={"ID":"c82e4ac3-b83f-478d-94a6-104868225497","Type":"ContainerDied","Data":"7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58"} Jul 19 15:29:24.469286 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.469188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88" event={"ID":"c82e4ac3-b83f-478d-94a6-104868225497","Type":"ContainerDied","Data":"9beffcdcc69aeeebb681393f69889b57a83e0bda5fe8bf87991e044ce57c9fd5"} Jul 19 15:29:24.469286 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.469219 2580 scope.go:117] "RemoveContainer" containerID="7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58" Jul 19 15:29:24.479180 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.479162 2580 scope.go:117] "RemoveContainer" containerID="7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58" Jul 19 15:29:24.479438 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:29:24.479421 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58\": container with ID starting with 7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58 not found: ID does not exist" containerID="7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58" Jul 19 15:29:24.479495 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.479447 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58"} err="failed to get container status \"7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58\": rpc error: code = NotFound desc = could not find container \"7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58\": container with ID starting with 7abbd9ba32db23db7004e436059cf1fa8a97a248331fabc043215287277def58 not found: ID does not exist" Jul 19 15:29:24.494960 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.492115 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88"] Jul 19 15:29:24.499609 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:24.499573 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e7bec00b-74d9d8d785-jtf88"] Jul 19 15:29:25.663271 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:25.663238 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c82e4ac3-b83f-478d-94a6-104868225497" path="/var/lib/kubelet/pods/c82e4ac3-b83f-478d-94a6-104868225497/volumes" Jul 19 15:29:28.213356 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:28.213321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:28.369019 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:28.368987 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:28.606486 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:28.606465 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:29.078464 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:29.078380 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:29.180441 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:29.180405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:29.279249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:29.279217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:30.754840 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:30.754791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:36.568566 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:36.568519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:36.674115 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:36.674081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:36.768762 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:36.768719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:36.868861 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:36.868834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:29:37.261500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:29:37.261384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:03.245787 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:03.245750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:09.239577 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:09.239539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:09.341056 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:09.341023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:09.439174 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:09.439136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:16.454594 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:16.454537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:16.503119 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:16.503091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:25.340532 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:25.340445 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:30.988977 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:30.988938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:31.120994 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:31.120956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:31.245883 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:31.245799 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:41.742633 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:41.742576 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:42.548504 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:42.548466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:49.473682 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:49.473620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:49.580310 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:49.580274 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:49.673638 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:49.673577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:49.774476 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:49.774390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:50.535830 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:50.535793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:50.635424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:50.635390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:50.736855 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:50.736823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:52.368298 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:52.368264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:58.231450 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:58.231414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:58.297607 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:58.297543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:30:58.901720 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:30:58.901689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:04.634780 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:04.634748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:05.590255 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:05.590217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:08.474962 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:08.474909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:08.577857 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:08.577822 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:08.745431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:08.745347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:08.844287 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:08.844252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:09.330366 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:09.330329 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:09.430369 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:09.430336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:12.972838 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:12.972802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:13.642236 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:13.642199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:13.740237 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:13.740205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:14.116362 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:14.116339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:19.947670 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:19.947636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:20.012093 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:20.012065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:30.717197 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:30.717161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:35.379423 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:35.379386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:35.769923 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:35.769832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:41.470535 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:41.470498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:41.532214 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:41.532186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:41.735249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:41.735159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:41.958530 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:41.958495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:42.077351 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:42.077268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:42.176069 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:42.176034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:42.304720 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:42.304688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:47.963179 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:47.963147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:48.069107 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:48.069074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:48.164817 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:48.164783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:48.341859 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:48.341772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:53.935502 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:53.935417 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:31:54.041335 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:31:54.041298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:00.709340 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:00.709304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:01.014683 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:01.014596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:01.304199 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:01.304109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:03.689336 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:03.689302 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:03.791767 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:03.791727 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:03.945666 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:03.945555 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:04.331037 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:04.330950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:04.473301 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:04.473251 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:04.574071 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:04.574035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:08.222238 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:08.222194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:16.292497 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:16.292455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:32:29.796294 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:29.796267 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:32:29.810642 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:29.810611 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:32:29.814740 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:29.814717 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:32:29.821024 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:29.821006 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:32:46.724286 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.724245 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb"] Jul 19 15:32:46.724900 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.724880 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82e4ac3-b83f-478d-94a6-104868225497" containerName="maas-api" Jul 19 15:32:46.724989 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.724902 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82e4ac3-b83f-478d-94a6-104868225497" containerName="maas-api" Jul 19 15:32:46.725047 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.725008 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82e4ac3-b83f-478d-94a6-104868225497" containerName="maas-api" Jul 19 15:32:46.728275 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.728253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:46.730807 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.730783 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:32:46.734564 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.734537 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb"] Jul 19 15:32:46.737779 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.737756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433ba31d-92b1-4093-ae46-4732cb303f39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb\" (UID: \"433ba31d-92b1-4093-ae46-4732cb303f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:46.838884 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.838844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433ba31d-92b1-4093-ae46-4732cb303f39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb\" (UID: \"433ba31d-92b1-4093-ae46-4732cb303f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:46.839495 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:46.839470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433ba31d-92b1-4093-ae46-4732cb303f39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb\" (UID: \"433ba31d-92b1-4093-ae46-4732cb303f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:47.043076 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:47.042978 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:47.182272 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:47.182246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb"] Jul 19 15:32:47.183923 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:32:47.183893 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ba31d_92b1_4093_ae46_4732cb303f39.slice/crio-dc41c674ba009f577f9e7676ef5fe8185fe474a2f614ddd05310f60bb79e0cd6 WatchSource:0}: Error finding container dc41c674ba009f577f9e7676ef5fe8185fe474a2f614ddd05310f60bb79e0cd6: Status 404 returned error can't find the container with id dc41c674ba009f577f9e7676ef5fe8185fe474a2f614ddd05310f60bb79e0cd6 Jul 19 15:32:47.185830 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:47.185815 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:32:47.633825 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:47.633791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" event={"ID":"433ba31d-92b1-4093-ae46-4732cb303f39","Type":"ContainerStarted","Data":"dc41c674ba009f577f9e7676ef5fe8185fe474a2f614ddd05310f60bb79e0cd6"} Jul 19 15:32:50.655097 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:50.655056 2580 generic.go:356] "Generic (PLEG): container finished" podID="433ba31d-92b1-4093-ae46-4732cb303f39" containerID="790fdafbb7e51b2b537da4978ce28a46bf29fe3b7ac686c8712469775e2f08a8" exitCode=0 Jul 19 15:32:50.655480 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:50.655143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" event={"ID":"433ba31d-92b1-4093-ae46-4732cb303f39","Type":"ContainerDied","Data":"790fdafbb7e51b2b537da4978ce28a46bf29fe3b7ac686c8712469775e2f08a8"} Jul 19 15:32:51.798373 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:51.798348 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:51.894646 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:51.894615 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433ba31d-92b1-4093-ae46-4732cb303f39-maas-api-service-ca\") pod \"433ba31d-92b1-4093-ae46-4732cb303f39\" (UID: \"433ba31d-92b1-4093-ae46-4732cb303f39\") " Jul 19 15:32:51.895010 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:51.894988 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433ba31d-92b1-4093-ae46-4732cb303f39-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "433ba31d-92b1-4093-ae46-4732cb303f39" (UID: "433ba31d-92b1-4093-ae46-4732cb303f39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:32:51.995523 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:51.995434 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/433ba31d-92b1-4093-ae46-4732cb303f39-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:32:52.668470 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:52.668442 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" Jul 19 15:32:52.668672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:52.668485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb" event={"ID":"433ba31d-92b1-4093-ae46-4732cb303f39","Type":"ContainerDied","Data":"dc41c674ba009f577f9e7676ef5fe8185fe474a2f614ddd05310f60bb79e0cd6"} Jul 19 15:32:52.668672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:32:52.668531 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc41c674ba009f577f9e7676ef5fe8185fe474a2f614ddd05310f60bb79e0cd6" Jul 19 15:33:02.125866 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:02.125825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:06.794455 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:06.794420 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb"] Jul 19 15:33:06.797110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:06.797084 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cc1f323c-6j4fb"] Jul 19 15:33:06.799321 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:06.799299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:07.662992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:07.662959 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433ba31d-92b1-4093-ae46-4732cb303f39" path="/var/lib/kubelet/pods/433ba31d-92b1-4093-ae46-4732cb303f39/volumes" Jul 19 15:33:09.021430 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:09.021393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:14.045801 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:14.045762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:24.656175 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:24.656062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:24.755162 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:24.755111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:43.613936 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:43.613887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:43.667782 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:43.667745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:46.525692 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:46.525661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:47.270039 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:47.269989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:52.712039 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:52.712007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:52.777380 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:52.777343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:52.986518 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:52.986431 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:53.226380 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:53.226350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:54.299044 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:54.298992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:54.392014 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:54.391982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:54.491179 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:54.491142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:55.261814 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:55.261782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:55.405814 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:55.405783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:33:56.799992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:33:56.799958 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:02.444634 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:02.444597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:02.507650 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:02.507617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:03.099418 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:03.099389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:08.878334 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:08.878293 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:08.983290 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:08.983261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:09.079238 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:09.079204 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:19.600845 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:19.600807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:19.928855 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:19.928821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:40.897091 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:40.897054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:47.270247 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:47.270211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:47.370714 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:47.370681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:47.611391 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:47.611352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:47.920993 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:47.920907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:48.014915 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:48.014877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:50.459948 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:50.459912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:50.534418 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:50.534386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:50.610971 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:50.610932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:50.925435 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:50.925402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:51.205014 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:51.204929 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:51.304471 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:51.304439 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:54.955474 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:54.955397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:55.289250 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:55.289168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:55.584470 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:55.584390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:56.156683 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:56.156651 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:58.554471 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:58.554435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:58.657053 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:58.657016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:58.855970 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:58.855888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:59.128755 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:59.128727 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:59.417939 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:59.417852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:34:59.518563 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:34:59.518529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:02.994355 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:02.994323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:12.832599 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:12.832544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:12.928851 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:12.928817 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:13.030266 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:13.030232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:20.144878 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:20.144832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:20.245662 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:20.245634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:36.532212 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:36.532178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:56.427646 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:56.427614 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:57.339839 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.339807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:57.475758 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.475725 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h"] Jul 19 15:35:57.476152 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.476134 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433ba31d-92b1-4093-ae46-4732cb303f39" containerName="revoke-keys" Jul 19 15:35:57.476152 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.476148 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="433ba31d-92b1-4093-ae46-4732cb303f39" containerName="revoke-keys" Jul 19 15:35:57.476230 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.476226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="433ba31d-92b1-4093-ae46-4732cb303f39" containerName="revoke-keys" Jul 19 15:35:57.479406 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.479390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:35:57.481620 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.481597 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:35:57.485542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.485522 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h"] Jul 19 15:35:57.519078 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.519052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc856600-66b8-49f4-92a5-ed6bb406c70c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h\" (UID: \"fc856600-66b8-49f4-92a5-ed6bb406c70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:35:57.620141 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.620108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc856600-66b8-49f4-92a5-ed6bb406c70c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h\" (UID: \"fc856600-66b8-49f4-92a5-ed6bb406c70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:35:57.620775 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.620755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc856600-66b8-49f4-92a5-ed6bb406c70c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h\" (UID: \"fc856600-66b8-49f4-92a5-ed6bb406c70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:35:57.790703 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.790671 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:35:57.975786 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:57.975748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:58.078776 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:58.078742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:35:58.140224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:58.140201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h"] Jul 19 15:35:58.141434 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:35:58.141405 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc856600_66b8_49f4_92a5_ed6bb406c70c.slice/crio-0c308df78285ebbe63a0183385de69b0df0dbcc11c864e86fc6b8a3a1078a491 WatchSource:0}: Error finding container 0c308df78285ebbe63a0183385de69b0df0dbcc11c864e86fc6b8a3a1078a491: Status 404 returned error can't find the container with id 0c308df78285ebbe63a0183385de69b0df0dbcc11c864e86fc6b8a3a1078a491 Jul 19 15:35:58.748032 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:58.747946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" event={"ID":"fc856600-66b8-49f4-92a5-ed6bb406c70c","Type":"ContainerStarted","Data":"d7a3110cd8b2a80ba19e091414fbbfc80620db4796ae8ac8430e8a0b6085e468"} Jul 19 15:35:58.748032 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:58.748003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" event={"ID":"fc856600-66b8-49f4-92a5-ed6bb406c70c","Type":"ContainerStarted","Data":"0c308df78285ebbe63a0183385de69b0df0dbcc11c864e86fc6b8a3a1078a491"} Jul 19 15:35:58.763735 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:58.763684 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" podStartSLOduration=1.763669449 podStartE2EDuration="1.763669449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:35:58.762088698 +0000 UTC m=+2309.626363494" watchObservedRunningTime="2026-07-19 15:35:58.763669449 +0000 UTC m=+2309.627944300" Jul 19 15:35:59.754938 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:59.754904 2580 generic.go:356] "Generic (PLEG): container finished" podID="fc856600-66b8-49f4-92a5-ed6bb406c70c" containerID="d7a3110cd8b2a80ba19e091414fbbfc80620db4796ae8ac8430e8a0b6085e468" exitCode=0 Jul 19 15:35:59.755329 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:35:59.754992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" event={"ID":"fc856600-66b8-49f4-92a5-ed6bb406c70c","Type":"ContainerDied","Data":"d7a3110cd8b2a80ba19e091414fbbfc80620db4796ae8ac8430e8a0b6085e468"} Jul 19 15:36:00.901080 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:00.901056 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:36:00.950357 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:00.950329 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc856600-66b8-49f4-92a5-ed6bb406c70c-maas-api-service-ca\") pod \"fc856600-66b8-49f4-92a5-ed6bb406c70c\" (UID: \"fc856600-66b8-49f4-92a5-ed6bb406c70c\") " Jul 19 15:36:00.950676 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:00.950652 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc856600-66b8-49f4-92a5-ed6bb406c70c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc856600-66b8-49f4-92a5-ed6bb406c70c" (UID: "fc856600-66b8-49f4-92a5-ed6bb406c70c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:36:01.052029 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:01.051948 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc856600-66b8-49f4-92a5-ed6bb406c70c-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:36:01.768288 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:01.768252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" event={"ID":"fc856600-66b8-49f4-92a5-ed6bb406c70c","Type":"ContainerDied","Data":"0c308df78285ebbe63a0183385de69b0df0dbcc11c864e86fc6b8a3a1078a491"} Jul 19 15:36:01.768288 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:01.768289 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h" Jul 19 15:36:01.768517 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:01.768301 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c308df78285ebbe63a0183385de69b0df0dbcc11c864e86fc6b8a3a1078a491" Jul 19 15:36:07.912466 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:07.912437 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:12.543431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:12.543398 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:12.552969 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:12.552930 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h"] Jul 19 15:36:12.555499 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:12.555478 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1dc017-rzk7h"] Jul 19 15:36:12.910816 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:12.910772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:13.662709 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:13.662676 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc856600-66b8-49f4-92a5-ed6bb406c70c" path="/var/lib/kubelet/pods/fc856600-66b8-49f4-92a5-ed6bb406c70c/volumes" Jul 19 15:36:18.280883 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:18.280847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:18.329943 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:18.329913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:18.750294 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:18.750262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:18.809179 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:18.809151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:19.004309 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.004219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:19.237807 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.237777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:19.354717 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.354638 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw"] Jul 19 15:36:19.355088 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.355075 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc856600-66b8-49f4-92a5-ed6bb406c70c" containerName="revoke-keys" Jul 19 15:36:19.355088 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.355089 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc856600-66b8-49f4-92a5-ed6bb406c70c" containerName="revoke-keys" Jul 19 15:36:19.355165 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.355146 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc856600-66b8-49f4-92a5-ed6bb406c70c" containerName="revoke-keys" Jul 19 15:36:19.358385 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.358368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:19.360549 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.360527 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:36:19.364627 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.364574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw"] Jul 19 15:36:19.430278 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.430242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf6a551-098c-498a-a1b1-29cb28cc85f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw\" (UID: \"0bf6a551-098c-498a-a1b1-29cb28cc85f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:19.531470 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.531434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf6a551-098c-498a-a1b1-29cb28cc85f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw\" (UID: \"0bf6a551-098c-498a-a1b1-29cb28cc85f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:19.532075 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.532056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf6a551-098c-498a-a1b1-29cb28cc85f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw\" (UID: \"0bf6a551-098c-498a-a1b1-29cb28cc85f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:19.671624 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.671573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:19.812175 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.812141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw"] Jul 19 15:36:19.813703 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:36:19.813679 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf6a551_098c_498a_a1b1_29cb28cc85f8.slice/crio-bd924676a4fdba91d314c8281f23c199e11a677144d8cea87a93ef31d39fe81c WatchSource:0}: Error finding container bd924676a4fdba91d314c8281f23c199e11a677144d8cea87a93ef31d39fe81c: Status 404 returned error can't find the container with id bd924676a4fdba91d314c8281f23c199e11a677144d8cea87a93ef31d39fe81c Jul 19 15:36:19.877404 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.877378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" event={"ID":"0bf6a551-098c-498a-a1b1-29cb28cc85f8","Type":"ContainerStarted","Data":"bd924676a4fdba91d314c8281f23c199e11a677144d8cea87a93ef31d39fe81c"} Jul 19 15:36:19.933789 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:19.933706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:20.034367 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:20.034334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:20.884479 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:20.884445 2580 generic.go:356] "Generic (PLEG): container finished" podID="0bf6a551-098c-498a-a1b1-29cb28cc85f8" containerID="61f45b9795b7745efacab17fbfde193a99d49dd2c54aa0fb2b1f44daf42af2e2" exitCode=0 Jul 19 15:36:20.884887 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:20.884537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" event={"ID":"0bf6a551-098c-498a-a1b1-29cb28cc85f8","Type":"ContainerDied","Data":"61f45b9795b7745efacab17fbfde193a99d49dd2c54aa0fb2b1f44daf42af2e2"} Jul 19 15:36:22.031571 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.031547 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:22.156354 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.156263 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf6a551-098c-498a-a1b1-29cb28cc85f8-maas-api-service-ca\") pod \"0bf6a551-098c-498a-a1b1-29cb28cc85f8\" (UID: \"0bf6a551-098c-498a-a1b1-29cb28cc85f8\") " Jul 19 15:36:22.156697 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.156676 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf6a551-098c-498a-a1b1-29cb28cc85f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bf6a551-098c-498a-a1b1-29cb28cc85f8" (UID: "0bf6a551-098c-498a-a1b1-29cb28cc85f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:36:22.257805 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.257769 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf6a551-098c-498a-a1b1-29cb28cc85f8-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:36:22.897478 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.897447 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" Jul 19 15:36:22.897478 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.897464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw" event={"ID":"0bf6a551-098c-498a-a1b1-29cb28cc85f8","Type":"ContainerDied","Data":"bd924676a4fdba91d314c8281f23c199e11a677144d8cea87a93ef31d39fe81c"} Jul 19 15:36:22.897734 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:22.897506 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd924676a4fdba91d314c8281f23c199e11a677144d8cea87a93ef31d39fe81c" Jul 19 15:36:23.748575 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:23.748540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:29.773832 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:29.773798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:34.425709 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:34.425674 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw"] Jul 19 15:36:34.428130 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:34.428107 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5e6749-qvsdw"] Jul 19 15:36:34.431311 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:34.431280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:34.838938 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:34.838848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:35.663138 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:35.663108 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf6a551-098c-498a-a1b1-29cb28cc85f8" path="/var/lib/kubelet/pods/0bf6a551-098c-498a-a1b1-29cb28cc85f8/volumes" Jul 19 15:36:40.575127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:40.575096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:40.632232 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:40.632202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:40.841202 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:40.841103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:41.073243 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:41.073210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:41.483334 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:41.483291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:41.783952 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:41.783872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:42.153896 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:42.153859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:45.057271 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:45.057237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:45.149992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:45.149955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:45.339906 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:45.339828 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:45.462481 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:45.462449 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:45.885015 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:45.884984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:45.984336 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:45.984282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:49.606770 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:49.606735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:50.082748 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:50.082657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:50.388744 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:50.388716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:50.497810 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:50.497767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:53.059089 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.059044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:53.139169 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.139138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:53.336657 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.336550 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c"] Jul 19 15:36:53.337258 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.337233 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf6a551-098c-498a-a1b1-29cb28cc85f8" containerName="revoke-keys" Jul 19 15:36:53.337258 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.337256 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf6a551-098c-498a-a1b1-29cb28cc85f8" containerName="revoke-keys" Jul 19 15:36:53.337448 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.337393 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf6a551-098c-498a-a1b1-29cb28cc85f8" containerName="revoke-keys" Jul 19 15:36:53.340840 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.340818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.342999 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.342979 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b32976-serving-cert\"" Jul 19 15:36:53.353630 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.353607 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c"] Jul 19 15:36:53.372380 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.372340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:53.447806 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.447776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-openshift-service-ca\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.448006 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.447815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-maas-api-tls\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.448006 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.447988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmw65\" (UniqueName: \"kubernetes.io/projected/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-kube-api-access-fmw65\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.535425 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.535392 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:53.548642 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.548621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-openshift-service-ca\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.548767 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.548654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-maas-api-tls\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.548767 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.548717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmw65\" (UniqueName: \"kubernetes.io/projected/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-kube-api-access-fmw65\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.549385 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.549367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-openshift-service-ca\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.551306 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.551283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-maas-api-tls\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.556907 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.556884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmw65\" (UniqueName: \"kubernetes.io/projected/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-kube-api-access-fmw65\") pod \"maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.653379 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.653339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:53.791561 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.791530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c"] Jul 19 15:36:53.793128 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:36:53.793098 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9787a3e_7c7b_4ca4_ad5d_1ee56411d43a.slice/crio-33318c012dd69f54e95351d43b40a6127059a0869b31396deedfac50ef854fa0 WatchSource:0}: Error finding container 33318c012dd69f54e95351d43b40a6127059a0869b31396deedfac50ef854fa0: Status 404 returned error can't find the container with id 33318c012dd69f54e95351d43b40a6127059a0869b31396deedfac50ef854fa0 Jul 19 15:36:53.845943 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.845890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:53.946430 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:53.946344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:36:54.082812 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:54.082779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" event={"ID":"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a","Type":"ContainerStarted","Data":"33318c012dd69f54e95351d43b40a6127059a0869b31396deedfac50ef854fa0"} Jul 19 15:36:55.089875 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:55.089835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" event={"ID":"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a","Type":"ContainerStarted","Data":"4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749"} Jul 19 15:36:55.090345 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:55.089946 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:36:55.111948 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:55.111894 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" podStartSLOduration=1.731685464 podStartE2EDuration="2.111879742s" totalImagesPullingTime="380.194278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:36:55.11025169 +0000 UTC m=+2365.974526488" watchObservedRunningTime="2026-07-19 15:36:55.111879742 +0000 UTC m=+2365.976154539" Jul 19 15:36:57.570050 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:36:57.570019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:01.100406 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:01.100378 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:37:05.576990 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.576955 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr"] Jul 19 15:37:05.580355 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.580339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:05.582468 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.582446 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:37:05.586526 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.586500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr"] Jul 19 15:37:05.673539 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.673507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf599378-76c1-4455-a641-3306aca24d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr\" (UID: \"bf599378-76c1-4455-a641-3306aca24d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:05.774868 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.774833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf599378-76c1-4455-a641-3306aca24d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr\" (UID: \"bf599378-76c1-4455-a641-3306aca24d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:05.775481 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.775461 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf599378-76c1-4455-a641-3306aca24d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr\" (UID: \"bf599378-76c1-4455-a641-3306aca24d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:05.893500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:05.893467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:06.038989 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:06.038963 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr"] Jul 19 15:37:06.041058 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:37:06.041006 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf599378_76c1_4455_a641_3306aca24d04.slice/crio-9a28291778de58434a0bf4fa97de33a34d4707b1b847ad17db4197fe9cd490ce WatchSource:0}: Error finding container 9a28291778de58434a0bf4fa97de33a34d4707b1b847ad17db4197fe9cd490ce: Status 404 returned error can't find the container with id 9a28291778de58434a0bf4fa97de33a34d4707b1b847ad17db4197fe9cd490ce Jul 19 15:37:06.153841 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:06.153752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" event={"ID":"bf599378-76c1-4455-a641-3306aca24d04","Type":"ContainerStarted","Data":"9a28291778de58434a0bf4fa97de33a34d4707b1b847ad17db4197fe9cd490ce"} Jul 19 15:37:06.230486 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:06.230456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:06.294359 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:06.294318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:06.874198 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:06.874167 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:07.161099 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:07.161067 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf599378-76c1-4455-a641-3306aca24d04" containerID="7f0008740f79640324a13986ab10f677f8488e404c2a5d989f30fe7c9d7e0f80" exitCode=0 Jul 19 15:37:07.161282 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:07.161143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" event={"ID":"bf599378-76c1-4455-a641-3306aca24d04","Type":"ContainerDied","Data":"7f0008740f79640324a13986ab10f677f8488e404c2a5d989f30fe7c9d7e0f80"} Jul 19 15:37:08.309382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:08.309357 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:08.399046 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:08.399012 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf599378-76c1-4455-a641-3306aca24d04-maas-api-service-ca\") pod \"bf599378-76c1-4455-a641-3306aca24d04\" (UID: \"bf599378-76c1-4455-a641-3306aca24d04\") " Jul 19 15:37:08.399388 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:08.399360 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf599378-76c1-4455-a641-3306aca24d04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf599378-76c1-4455-a641-3306aca24d04" (UID: "bf599378-76c1-4455-a641-3306aca24d04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:37:08.500187 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:08.500102 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf599378-76c1-4455-a641-3306aca24d04-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:37:09.174328 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:09.174298 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" Jul 19 15:37:09.174534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:09.174335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr" event={"ID":"bf599378-76c1-4455-a641-3306aca24d04","Type":"ContainerDied","Data":"9a28291778de58434a0bf4fa97de33a34d4707b1b847ad17db4197fe9cd490ce"} Jul 19 15:37:09.174534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:09.174382 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a28291778de58434a0bf4fa97de33a34d4707b1b847ad17db4197fe9cd490ce" Jul 19 15:37:15.535252 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:15.535221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:15.636078 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:15.636048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:20.832448 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:20.832414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:25.655233 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:25.655197 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr"] Jul 19 15:37:25.656997 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:25.656970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:25.662290 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:25.662270 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c4f7c-zlrnr"] Jul 19 15:37:26.054650 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:26.054549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:27.663101 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:27.663062 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf599378-76c1-4455-a641-3306aca24d04" path="/var/lib/kubelet/pods/bf599378-76c1-4455-a641-3306aca24d04/volumes" Jul 19 15:37:29.845188 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:29.845158 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:37:29.851521 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:29.851498 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:37:29.864698 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:29.864671 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:37:29.870925 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:29.870902 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:37:31.659920 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:31.659888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:31.713716 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:31.713681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:31.926137 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:31.926049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:32.161950 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.161917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:32.285945 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.285856 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8"] Jul 19 15:37:32.286305 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.286292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf599378-76c1-4455-a641-3306aca24d04" containerName="revoke-keys" Jul 19 15:37:32.286347 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.286307 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf599378-76c1-4455-a641-3306aca24d04" containerName="revoke-keys" Jul 19 15:37:32.286382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.286376 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf599378-76c1-4455-a641-3306aca24d04" containerName="revoke-keys" Jul 19 15:37:32.290862 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.290845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:32.293344 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.293311 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:37:32.296358 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.296333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8"] Jul 19 15:37:32.329128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.329101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db91c876-f8b1-44c3-9de7-4914ca643e4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b32976-5l5f8\" (UID: \"db91c876-f8b1-44c3-9de7-4914ca643e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:32.429939 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.429893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db91c876-f8b1-44c3-9de7-4914ca643e4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b32976-5l5f8\" (UID: \"db91c876-f8b1-44c3-9de7-4914ca643e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:32.430542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.430522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db91c876-f8b1-44c3-9de7-4914ca643e4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b32976-5l5f8\" (UID: \"db91c876-f8b1-44c3-9de7-4914ca643e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:32.602781 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.602692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:32.739333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:32.739304 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8"] Jul 19 15:37:32.740148 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:37:32.740123 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb91c876_f8b1_44c3_9de7_4914ca643e4a.slice/crio-f70d36da4448eb11b0e04e76ae7b648d76c660520530eb1819df1c4afae38612 WatchSource:0}: Error finding container f70d36da4448eb11b0e04e76ae7b648d76c660520530eb1819df1c4afae38612: Status 404 returned error can't find the container with id f70d36da4448eb11b0e04e76ae7b648d76c660520530eb1819df1c4afae38612 Jul 19 15:37:33.321400 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:33.321306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" event={"ID":"db91c876-f8b1-44c3-9de7-4914ca643e4a","Type":"ContainerStarted","Data":"810154223d2d8263d5b68899346d414c6c9dcc4e461e2afdd69b4757629f4143"} Jul 19 15:37:33.321400 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:33.321371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" event={"ID":"db91c876-f8b1-44c3-9de7-4914ca643e4a","Type":"ContainerStarted","Data":"f70d36da4448eb11b0e04e76ae7b648d76c660520530eb1819df1c4afae38612"} Jul 19 15:37:33.338864 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:33.338816 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" podStartSLOduration=1.338801967 podStartE2EDuration="1.338801967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:37:33.335927626 +0000 UTC m=+2404.200202436" watchObservedRunningTime="2026-07-19 15:37:33.338801967 +0000 UTC m=+2404.203076797" Jul 19 15:37:34.328992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:34.328951 2580 generic.go:356] "Generic (PLEG): container finished" podID="db91c876-f8b1-44c3-9de7-4914ca643e4a" containerID="810154223d2d8263d5b68899346d414c6c9dcc4e461e2afdd69b4757629f4143" exitCode=0 Jul 19 15:37:34.329352 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:34.329022 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" event={"ID":"db91c876-f8b1-44c3-9de7-4914ca643e4a","Type":"ContainerDied","Data":"810154223d2d8263d5b68899346d414c6c9dcc4e461e2afdd69b4757629f4143"} Jul 19 15:37:35.472836 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:35.472810 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:35.558981 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:35.558952 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db91c876-f8b1-44c3-9de7-4914ca643e4a-maas-api-service-ca\") pod \"db91c876-f8b1-44c3-9de7-4914ca643e4a\" (UID: \"db91c876-f8b1-44c3-9de7-4914ca643e4a\") " Jul 19 15:37:35.559320 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:35.559293 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db91c876-f8b1-44c3-9de7-4914ca643e4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db91c876-f8b1-44c3-9de7-4914ca643e4a" (UID: "db91c876-f8b1-44c3-9de7-4914ca643e4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:37:35.662274 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:35.662236 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db91c876-f8b1-44c3-9de7-4914ca643e4a-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:37:36.343039 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:36.343004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" event={"ID":"db91c876-f8b1-44c3-9de7-4914ca643e4a","Type":"ContainerDied","Data":"f70d36da4448eb11b0e04e76ae7b648d76c660520530eb1819df1c4afae38612"} Jul 19 15:37:36.343230 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:36.343065 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f70d36da4448eb11b0e04e76ae7b648d76c660520530eb1819df1c4afae38612" Jul 19 15:37:36.343230 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:36.343017 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8" Jul 19 15:37:47.744268 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:47.744229 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c"] Jul 19 15:37:47.744815 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:47.744524 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" podUID="b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" containerName="maas-api" containerID="cri-o://4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749" gracePeriod=30 Jul 19 15:37:47.767156 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:47.767131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:47.998498 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:47.998427 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:37:48.076188 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.076157 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-openshift-service-ca\") pod \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " Jul 19 15:37:48.076407 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.076203 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-maas-api-tls\") pod \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " Jul 19 15:37:48.076407 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.076226 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmw65\" (UniqueName: \"kubernetes.io/projected/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-kube-api-access-fmw65\") pod \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\" (UID: \"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a\") " Jul 19 15:37:48.076553 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.076526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" (UID: "b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:37:48.078319 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.078287 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" (UID: "b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:37:48.078434 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.078327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-kube-api-access-fmw65" (OuterVolumeSpecName: "kube-api-access-fmw65") pod "b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" (UID: "b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a"). InnerVolumeSpecName "kube-api-access-fmw65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:37:48.177237 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.177204 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-openshift-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:37:48.177237 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.177231 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-maas-api-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:37:48.177237 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.177241 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmw65\" (UniqueName: \"kubernetes.io/projected/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a-kube-api-access-fmw65\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:37:48.423865 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.423821 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" containerID="4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749" exitCode=0 Jul 19 15:37:48.424042 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.423878 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" Jul 19 15:37:48.424042 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.423880 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" event={"ID":"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a","Type":"ContainerDied","Data":"4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749"} Jul 19 15:37:48.424042 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.423957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c" event={"ID":"b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a","Type":"ContainerDied","Data":"33318c012dd69f54e95351d43b40a6127059a0869b31396deedfac50ef854fa0"} Jul 19 15:37:48.424042 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.423981 2580 scope.go:117] "RemoveContainer" containerID="4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749" Jul 19 15:37:48.434380 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.434362 2580 scope.go:117] "RemoveContainer" containerID="4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749" Jul 19 15:37:48.434718 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:37:48.434696 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749\": container with ID starting with 4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749 not found: ID does not exist" containerID="4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749" Jul 19 15:37:48.434798 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.434727 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749"} err="failed to get container status \"4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749\": rpc error: code = NotFound desc = could not find container \"4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749\": container with ID starting with 4d235c321a5ab5b65ce0bf42823eefecc5d7275d6ef872e8a32c9efe76dad749 not found: ID does not exist" Jul 19 15:37:48.445871 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.445843 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c"] Jul 19 15:37:48.449426 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:48.449404 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b32976-58b6f58dbd-d8l9c"] Jul 19 15:37:49.664571 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:49.664537 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" path="/var/lib/kubelet/pods/b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a/volumes" Jul 19 15:37:52.362262 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:52.362167 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8"] Jul 19 15:37:52.363382 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:52.363360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:52.364805 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:52.364782 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b32976-5l5f8"] Jul 19 15:37:52.766531 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:52.766448 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:53.663954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:53.663920 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db91c876-f8b1-44c3-9de7-4914ca643e4a" path="/var/lib/kubelet/pods/db91c876-f8b1-44c3-9de7-4914ca643e4a/volumes" Jul 19 15:37:58.698146 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:58.698102 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:58.797842 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:58.797813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:58.974016 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:58.973930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:59.198218 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:59.198186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:59.537878 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:59.537839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:37:59.859919 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:37:59.859889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:00.009661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:00.009632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:02.367955 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:02.367918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:02.455400 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:02.455366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:02.523494 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:02.523460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:02.623877 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:02.623778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:02.928500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:02.928412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:03.029615 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:03.029568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:04.165185 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:04.165146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:06.880344 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:06.880299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:07.218117 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:07.218036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:07.518051 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:07.517965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:08.186418 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:08.186386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:10.379685 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:10.379646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:10.465548 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:10.465509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:10.540293 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:10.540265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:10.985911 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:10.985880 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:11.087262 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:11.087214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:11.187719 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:11.187686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:14.883038 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:14.883011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:23.653032 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:23.652996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:23.753934 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:23.753897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:25.399462 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:25.399428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:25.501992 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:25.501955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:27.224995 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:27.224958 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:31.773954 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:31.773907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:31.872470 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:31.872441 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:32.009493 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:32.009457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:38:32.063208 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:38:32.063126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:39:12.333671 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:39:12.333634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:39:19.489850 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:39:19.489818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:39:30.148636 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:39:30.148549 2580 scope.go:117] "RemoveContainer" containerID="790fdafbb7e51b2b537da4978ce28a46bf29fe3b7ac686c8712469775e2f08a8" Jul 19 15:41:18.294903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:41:18.294822 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:41:19.369137 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:41:19.369098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:41:30.323317 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:41:30.323264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:41:41.715229 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:41:41.715193 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:41:50.926461 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:41:50.926410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:42:29.887186 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:29.887156 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:42:29.894145 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:29.894121 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:42:29.907533 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:29.907490 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:42:29.913506 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:29.913487 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h4h72_88f133f4-9d7a-4622-8eb8-97c9a630fbde/ovn-acl-logging/0.log" Jul 19 15:42:30.187914 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:30.187832 2580 scope.go:117] "RemoveContainer" containerID="61f45b9795b7745efacab17fbfde193a99d49dd2c54aa0fb2b1f44daf42af2e2" Jul 19 15:42:30.196859 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:30.196840 2580 scope.go:117] "RemoveContainer" containerID="d7a3110cd8b2a80ba19e091414fbbfc80620db4796ae8ac8430e8a0b6085e468" Jul 19 15:42:33.712180 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:33.712148 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:42:34.719317 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:34.719288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:42:42.003089 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:42:42.003054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:18.424746 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:18.424712 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:23.919354 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:23.919321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:24.369420 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:24.369384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:24.469841 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:24.469808 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:26.113793 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:26.113761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:26.214429 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:26.214399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:30.221502 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:30.221465 2580 scope.go:117] "RemoveContainer" containerID="7f0008740f79640324a13986ab10f677f8488e404c2a5d989f30fe7c9d7e0f80" Jul 19 15:43:32.622192 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:32.622156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:43:32.720209 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:43:32.720150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:00.795330 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:00.795249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:01.697746 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:01.697702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:02.667572 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.667533 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7626-140573624565296"] Jul 19 15:44:02.668149 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.668128 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" containerName="maas-api" Jul 19 15:44:02.668149 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.668152 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" containerName="maas-api" Jul 19 15:44:02.668295 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.668190 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db91c876-f8b1-44c3-9de7-4914ca643e4a" containerName="revoke-keys" Jul 19 15:44:02.668295 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.668199 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="db91c876-f8b1-44c3-9de7-4914ca643e4a" containerName="revoke-keys" Jul 19 15:44:02.668400 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.668293 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="db91c876-f8b1-44c3-9de7-4914ca643e4a" containerName="revoke-keys" Jul 19 15:44:02.668400 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.668308 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9787a3e-7c7b-4ca4-ad5d-1ee56411d43a" containerName="maas-api" Jul 19 15:44:02.671531 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.671509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:02.674653 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.674620 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4vhrx\"" Jul 19 15:44:02.676966 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.676942 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573624565296"] Jul 19 15:44:02.774488 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.774345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbf2\" (UniqueName: \"kubernetes.io/projected/2eef4a54-e4ad-4738-a139-dd86956b0af4-kube-api-access-9mbf2\") pod \"test-curl-7626-140573624565296\" (UID: \"2eef4a54-e4ad-4738-a139-dd86956b0af4\") " pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:02.875507 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.875466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbf2\" (UniqueName: \"kubernetes.io/projected/2eef4a54-e4ad-4738-a139-dd86956b0af4-kube-api-access-9mbf2\") pod \"test-curl-7626-140573624565296\" (UID: \"2eef4a54-e4ad-4738-a139-dd86956b0af4\") " pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:02.883281 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.883249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbf2\" (UniqueName: \"kubernetes.io/projected/2eef4a54-e4ad-4738-a139-dd86956b0af4-kube-api-access-9mbf2\") pod \"test-curl-7626-140573624565296\" (UID: \"2eef4a54-e4ad-4738-a139-dd86956b0af4\") " pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:02.982913 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:02.982829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:03.119044 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:03.119016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573624565296"] Jul 19 15:44:03.121104 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:03.121076 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eef4a54_e4ad_4738_a139_dd86956b0af4.slice/crio-01620a8344cad0c4ebdb424785a1740f4e924badceb6ecf87063e4480fa14959 WatchSource:0}: Error finding container 01620a8344cad0c4ebdb424785a1740f4e924badceb6ecf87063e4480fa14959: Status 404 returned error can't find the container with id 01620a8344cad0c4ebdb424785a1740f4e924badceb6ecf87063e4480fa14959 Jul 19 15:44:03.122450 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:03.122434 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:44:03.663942 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:03.663900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573624565296" event={"ID":"2eef4a54-e4ad-4738-a139-dd86956b0af4","Type":"ContainerStarted","Data":"01620a8344cad0c4ebdb424785a1740f4e924badceb6ecf87063e4480fa14959"} Jul 19 15:44:04.665632 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:04.665577 2580 generic.go:356] "Generic (PLEG): container finished" podID="2eef4a54-e4ad-4738-a139-dd86956b0af4" containerID="f7476dbe206bcd679073ca5d093ccc74bb12d7996e2cd808f19a7d35c97b5723" exitCode=0 Jul 19 15:44:04.665632 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:04.665624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573624565296" event={"ID":"2eef4a54-e4ad-4738-a139-dd86956b0af4","Type":"ContainerDied","Data":"f7476dbe206bcd679073ca5d093ccc74bb12d7996e2cd808f19a7d35c97b5723"} Jul 19 15:44:05.819812 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.819785 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:05.839126 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.839093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7626-140573624565296_2eef4a54-e4ad-4738-a139-dd86956b0af4/test-curl-7626-140573624565296/0.log" Jul 19 15:44:05.866699 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.866665 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7626-140573624565296"] Jul 19 15:44:05.874014 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.873985 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7626-140573624565296"] Jul 19 15:44:05.906113 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.906084 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mbf2\" (UniqueName: \"kubernetes.io/projected/2eef4a54-e4ad-4738-a139-dd86956b0af4-kube-api-access-9mbf2\") pod \"2eef4a54-e4ad-4738-a139-dd86956b0af4\" (UID: \"2eef4a54-e4ad-4738-a139-dd86956b0af4\") " Jul 19 15:44:05.908225 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.908199 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eef4a54-e4ad-4738-a139-dd86956b0af4-kube-api-access-9mbf2" (OuterVolumeSpecName: "kube-api-access-9mbf2") pod "2eef4a54-e4ad-4738-a139-dd86956b0af4" (UID: "2eef4a54-e4ad-4738-a139-dd86956b0af4"). InnerVolumeSpecName "kube-api-access-9mbf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:05.987071 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.986988 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7626-140573624525488"] Jul 19 15:44:05.987451 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.987438 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eef4a54-e4ad-4738-a139-dd86956b0af4" containerName="test-curl-7626-140573624565296" Jul 19 15:44:05.987498 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.987453 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eef4a54-e4ad-4738-a139-dd86956b0af4" containerName="test-curl-7626-140573624565296" Jul 19 15:44:05.987543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.987534 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eef4a54-e4ad-4738-a139-dd86956b0af4" containerName="test-curl-7626-140573624565296" Jul 19 15:44:05.990691 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.990673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:05.996051 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:05.996027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573624525488"] Jul 19 15:44:06.007072 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.007043 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbf2\" (UniqueName: \"kubernetes.io/projected/2eef4a54-e4ad-4738-a139-dd86956b0af4-kube-api-access-9mbf2\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:06.108456 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.108414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btjrp\" (UniqueName: \"kubernetes.io/projected/6216e06c-c9a6-48a8-973b-5b534cee8361-kube-api-access-btjrp\") pod \"test-curl-7626-140573624525488\" (UID: \"6216e06c-c9a6-48a8-973b-5b534cee8361\") " pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:06.209060 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.209020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btjrp\" (UniqueName: \"kubernetes.io/projected/6216e06c-c9a6-48a8-973b-5b534cee8361-kube-api-access-btjrp\") pod \"test-curl-7626-140573624525488\" (UID: \"6216e06c-c9a6-48a8-973b-5b534cee8361\") " pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:06.216646 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.216615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btjrp\" (UniqueName: \"kubernetes.io/projected/6216e06c-c9a6-48a8-973b-5b534cee8361-kube-api-access-btjrp\") pod \"test-curl-7626-140573624525488\" (UID: \"6216e06c-c9a6-48a8-973b-5b534cee8361\") " pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:06.302293 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.302205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:06.442172 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.442141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573624525488"] Jul 19 15:44:06.443675 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:06.443645 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6216e06c_c9a6_48a8_973b_5b534cee8361.slice/crio-a8950edb2cbdc173eb54300c2e4ef7ac0a26c78e422f297e51b23fa1875a8a4e WatchSource:0}: Error finding container a8950edb2cbdc173eb54300c2e4ef7ac0a26c78e422f297e51b23fa1875a8a4e: Status 404 returned error can't find the container with id a8950edb2cbdc173eb54300c2e4ef7ac0a26c78e422f297e51b23fa1875a8a4e Jul 19 15:44:06.677602 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.677553 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:06.677786 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.677611 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01620a8344cad0c4ebdb424785a1740f4e924badceb6ecf87063e4480fa14959" Jul 19 15:44:06.678944 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:06.678912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573624525488" event={"ID":"6216e06c-c9a6-48a8-973b-5b534cee8361","Type":"ContainerStarted","Data":"a8950edb2cbdc173eb54300c2e4ef7ac0a26c78e422f297e51b23fa1875a8a4e"} Jul 19 15:44:06.690133 ip-10-0-140-163 kubenswrapper[2580]: E0719 15:44:06.690104 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7626-140573624565296\" is forbidden: User \"system:node:ip-10-0-140-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-163.ec2.internal' and this object" podUID="2eef4a54-e4ad-4738-a139-dd86956b0af4" pod="opendatahub/test-curl-7626-140573624565296" Jul 19 15:44:07.665923 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:07.665884 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eef4a54-e4ad-4738-a139-dd86956b0af4" path="/var/lib/kubelet/pods/2eef4a54-e4ad-4738-a139-dd86956b0af4/volumes" Jul 19 15:44:07.687032 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:07.686997 2580 generic.go:356] "Generic (PLEG): container finished" podID="6216e06c-c9a6-48a8-973b-5b534cee8361" containerID="db28eef956cc30ba4e8c617e8be7ecc67eb2524287301a8e70431c71f74d92a8" exitCode=0 Jul 19 15:44:07.687370 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:07.687347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573624525488" event={"ID":"6216e06c-c9a6-48a8-973b-5b534cee8361","Type":"ContainerDied","Data":"db28eef956cc30ba4e8c617e8be7ecc67eb2524287301a8e70431c71f74d92a8"} Jul 19 15:44:08.835106 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:08.835081 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:08.854843 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:08.854813 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7626-140573624525488_6216e06c-c9a6-48a8-973b-5b534cee8361/test-curl-7626-140573624525488/0.log" Jul 19 15:44:08.880055 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:08.880010 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7626-140573624525488"] Jul 19 15:44:08.882558 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:08.882524 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7626-140573624525488"] Jul 19 15:44:08.935403 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:08.935375 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btjrp\" (UniqueName: \"kubernetes.io/projected/6216e06c-c9a6-48a8-973b-5b534cee8361-kube-api-access-btjrp\") pod \"6216e06c-c9a6-48a8-973b-5b534cee8361\" (UID: \"6216e06c-c9a6-48a8-973b-5b534cee8361\") " Jul 19 15:44:08.937322 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:08.937295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6216e06c-c9a6-48a8-973b-5b534cee8361-kube-api-access-btjrp" (OuterVolumeSpecName: "kube-api-access-btjrp") pod "6216e06c-c9a6-48a8-973b-5b534cee8361" (UID: "6216e06c-c9a6-48a8-973b-5b534cee8361"). InnerVolumeSpecName "kube-api-access-btjrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:09.037044 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.036951 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btjrp\" (UniqueName: \"kubernetes.io/projected/6216e06c-c9a6-48a8-973b-5b534cee8361-kube-api-access-btjrp\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:09.040697 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.040667 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:09.041095 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.041081 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6216e06c-c9a6-48a8-973b-5b534cee8361" containerName="test-curl-7626-140573624525488" Jul 19 15:44:09.041134 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.041097 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6216e06c-c9a6-48a8-973b-5b534cee8361" containerName="test-curl-7626-140573624525488" Jul 19 15:44:09.041196 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.041186 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6216e06c-c9a6-48a8-973b-5b534cee8361" containerName="test-curl-7626-140573624525488" Jul 19 15:44:09.044292 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.044276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:09.049921 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.049881 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:09.137899 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.137862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4msb5\" (UniqueName: \"kubernetes.io/projected/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4-kube-api-access-4msb5\") pod \"test-curl-7626-140573623272624\" (UID: \"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4\") " pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:09.239229 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.239196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4msb5\" (UniqueName: \"kubernetes.io/projected/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4-kube-api-access-4msb5\") pod \"test-curl-7626-140573623272624\" (UID: \"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4\") " pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:09.247199 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.247167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4msb5\" (UniqueName: \"kubernetes.io/projected/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4-kube-api-access-4msb5\") pod \"test-curl-7626-140573623272624\" (UID: \"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4\") " pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:09.356017 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.355917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:09.491996 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.491967 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:09.493794 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:09.493765 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09df8dc3_6cf9_4ed6_a295_c7c3719f91f4.slice/crio-0a8ecbd2d572dda58e16c3dcdc25a6b6d56dd6704ed4cb3a2e3bd7ad04d17d21 WatchSource:0}: Error finding container 0a8ecbd2d572dda58e16c3dcdc25a6b6d56dd6704ed4cb3a2e3bd7ad04d17d21: Status 404 returned error can't find the container with id 0a8ecbd2d572dda58e16c3dcdc25a6b6d56dd6704ed4cb3a2e3bd7ad04d17d21 Jul 19 15:44:09.663467 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.663430 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6216e06c-c9a6-48a8-973b-5b534cee8361" path="/var/lib/kubelet/pods/6216e06c-c9a6-48a8-973b-5b534cee8361/volumes" Jul 19 15:44:09.709162 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.709120 2580 scope.go:117] "RemoveContainer" containerID="db28eef956cc30ba4e8c617e8be7ecc67eb2524287301a8e70431c71f74d92a8" Jul 19 15:44:09.709162 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.709135 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573624525488" Jul 19 15:44:09.710708 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:09.710662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623272624" event={"ID":"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4","Type":"ContainerStarted","Data":"0a8ecbd2d572dda58e16c3dcdc25a6b6d56dd6704ed4cb3a2e3bd7ad04d17d21"} Jul 19 15:44:10.724258 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:10.724211 2580 generic.go:356] "Generic (PLEG): container finished" podID="09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" containerID="39d6a42a6239db94a2b3353d3f46e839cf89430ddb0f3b02613fa65dc4fc77a1" exitCode=0 Jul 19 15:44:10.724652 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:10.724291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623272624" event={"ID":"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4","Type":"ContainerDied","Data":"39d6a42a6239db94a2b3353d3f46e839cf89430ddb0f3b02613fa65dc4fc77a1"} Jul 19 15:44:11.877023 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:11.876999 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:11.896114 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:11.896086 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7626-140573623272624_09df8dc3-6cf9-4ed6-a295-c7c3719f91f4/test-curl-7626-140573623272624/0.log" Jul 19 15:44:11.919153 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:11.919123 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:11.922803 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:11.922778 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:11.965374 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:11.965344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4msb5\" (UniqueName: \"kubernetes.io/projected/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4-kube-api-access-4msb5\") pod \"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4\" (UID: \"09df8dc3-6cf9-4ed6-a295-c7c3719f91f4\") " Jul 19 15:44:11.967500 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:11.967474 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4-kube-api-access-4msb5" (OuterVolumeSpecName: "kube-api-access-4msb5") pod "09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" (UID: "09df8dc3-6cf9-4ed6-a295-c7c3719f91f4"). InnerVolumeSpecName "kube-api-access-4msb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:12.038846 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.038763 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:12.039274 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.039258 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" containerName="test-curl-7626-140573623272624" Jul 19 15:44:12.039317 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.039277 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" containerName="test-curl-7626-140573623272624" Jul 19 15:44:12.039374 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.039364 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" containerName="test-curl-7626-140573623272624" Jul 19 15:44:12.042505 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.042489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:12.048514 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.048489 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:12.066105 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.066074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqqh\" (UniqueName: \"kubernetes.io/projected/3010843e-c901-4ba4-ac05-ce1a4013ab1a-kube-api-access-fqqqh\") pod \"test-curl-7626-140573623272624\" (UID: \"3010843e-c901-4ba4-ac05-ce1a4013ab1a\") " pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:12.066249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.066235 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4msb5\" (UniqueName: \"kubernetes.io/projected/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4-kube-api-access-4msb5\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:12.167263 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.167228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqqh\" (UniqueName: \"kubernetes.io/projected/3010843e-c901-4ba4-ac05-ce1a4013ab1a-kube-api-access-fqqqh\") pod \"test-curl-7626-140573623272624\" (UID: \"3010843e-c901-4ba4-ac05-ce1a4013ab1a\") " pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:12.175238 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.175202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqqh\" (UniqueName: \"kubernetes.io/projected/3010843e-c901-4ba4-ac05-ce1a4013ab1a-kube-api-access-fqqqh\") pod \"test-curl-7626-140573623272624\" (UID: \"3010843e-c901-4ba4-ac05-ce1a4013ab1a\") " pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:12.353952 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.353863 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:12.488797 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.488769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:12.489410 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:12.489379 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3010843e_c901_4ba4_ac05_ce1a4013ab1a.slice/crio-f6b1f6ded204605a3c4e62f2bfc65105c5c98a7ac6aaba005d6bf586522e6510 WatchSource:0}: Error finding container f6b1f6ded204605a3c4e62f2bfc65105c5c98a7ac6aaba005d6bf586522e6510: Status 404 returned error can't find the container with id f6b1f6ded204605a3c4e62f2bfc65105c5c98a7ac6aaba005d6bf586522e6510 Jul 19 15:44:12.738998 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.738964 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623272624" event={"ID":"3010843e-c901-4ba4-ac05-ce1a4013ab1a","Type":"ContainerStarted","Data":"f6b1f6ded204605a3c4e62f2bfc65105c5c98a7ac6aaba005d6bf586522e6510"} Jul 19 15:44:12.740274 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.740254 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:12.740274 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.740269 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8ecbd2d572dda58e16c3dcdc25a6b6d56dd6704ed4cb3a2e3bd7ad04d17d21" Jul 19 15:44:12.751712 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:12.751678 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7626-140573623272624" oldPodUID="09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" podUID="3010843e-c901-4ba4-ac05-ce1a4013ab1a" Jul 19 15:44:13.663461 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:13.663425 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09df8dc3-6cf9-4ed6-a295-c7c3719f91f4" path="/var/lib/kubelet/pods/09df8dc3-6cf9-4ed6-a295-c7c3719f91f4/volumes" Jul 19 15:44:13.748206 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:13.748170 2580 generic.go:356] "Generic (PLEG): container finished" podID="3010843e-c901-4ba4-ac05-ce1a4013ab1a" containerID="35442fc1027190721599aa8ced87d0954b6b93e50b34bbd566eb0c0710ddb7c2" exitCode=0 Jul 19 15:44:13.748351 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:13.748252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623272624" event={"ID":"3010843e-c901-4ba4-ac05-ce1a4013ab1a","Type":"ContainerDied","Data":"35442fc1027190721599aa8ced87d0954b6b93e50b34bbd566eb0c0710ddb7c2"} Jul 19 15:44:14.900593 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:14.900555 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:14.920126 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:14.920094 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7626-140573623272624_3010843e-c901-4ba4-ac05-ce1a4013ab1a/test-curl-7626-140573623272624/0.log" Jul 19 15:44:14.946605 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:14.946555 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:14.949239 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:14.949219 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7626-140573623272624"] Jul 19 15:44:14.991183 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:14.991146 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqqqh\" (UniqueName: \"kubernetes.io/projected/3010843e-c901-4ba4-ac05-ce1a4013ab1a-kube-api-access-fqqqh\") pod \"3010843e-c901-4ba4-ac05-ce1a4013ab1a\" (UID: \"3010843e-c901-4ba4-ac05-ce1a4013ab1a\") " Jul 19 15:44:14.993330 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:14.993295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3010843e-c901-4ba4-ac05-ce1a4013ab1a-kube-api-access-fqqqh" (OuterVolumeSpecName: "kube-api-access-fqqqh") pod "3010843e-c901-4ba4-ac05-ce1a4013ab1a" (UID: "3010843e-c901-4ba4-ac05-ce1a4013ab1a"). InnerVolumeSpecName "kube-api-access-fqqqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:15.070824 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.070739 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:15.071283 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.071262 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3010843e-c901-4ba4-ac05-ce1a4013ab1a" containerName="test-curl-7626-140573623272624" Jul 19 15:44:15.071378 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.071285 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3010843e-c901-4ba4-ac05-ce1a4013ab1a" containerName="test-curl-7626-140573623272624" Jul 19 15:44:15.071430 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.071400 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3010843e-c901-4ba4-ac05-ce1a4013ab1a" containerName="test-curl-7626-140573623272624" Jul 19 15:44:15.074694 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.074662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:15.079701 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.079674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:15.092541 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.092511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmd5\" (UniqueName: \"kubernetes.io/projected/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e-kube-api-access-bwmd5\") pod \"test-curl-7626-140573623273632\" (UID: \"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e\") " pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:15.092705 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.092596 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqqqh\" (UniqueName: \"kubernetes.io/projected/3010843e-c901-4ba4-ac05-ce1a4013ab1a-kube-api-access-fqqqh\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:15.193808 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.193772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmd5\" (UniqueName: \"kubernetes.io/projected/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e-kube-api-access-bwmd5\") pod \"test-curl-7626-140573623273632\" (UID: \"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e\") " pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:15.202078 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.202054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmd5\" (UniqueName: \"kubernetes.io/projected/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e-kube-api-access-bwmd5\") pod \"test-curl-7626-140573623273632\" (UID: \"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e\") " pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:15.386242 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.386209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:15.526477 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.526452 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:15.528145 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:15.528120 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c56c4a1_2eef_4a8f_a34a_d2ab8b52c70e.slice/crio-88e105e60ee98d429509586aa74b101cf45211e8f40792d146df53297471b07f WatchSource:0}: Error finding container 88e105e60ee98d429509586aa74b101cf45211e8f40792d146df53297471b07f: Status 404 returned error can't find the container with id 88e105e60ee98d429509586aa74b101cf45211e8f40792d146df53297471b07f Jul 19 15:44:15.663158 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.663063 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3010843e-c901-4ba4-ac05-ce1a4013ab1a" path="/var/lib/kubelet/pods/3010843e-c901-4ba4-ac05-ce1a4013ab1a/volumes" Jul 19 15:44:15.760645 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.760618 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623272624" Jul 19 15:44:15.760811 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.760627 2580 scope.go:117] "RemoveContainer" containerID="35442fc1027190721599aa8ced87d0954b6b93e50b34bbd566eb0c0710ddb7c2" Jul 19 15:44:15.762100 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:15.762074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623273632" event={"ID":"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e","Type":"ContainerStarted","Data":"88e105e60ee98d429509586aa74b101cf45211e8f40792d146df53297471b07f"} Jul 19 15:44:16.770169 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:16.770128 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" containerID="7f4df7cf49ce64379a521d1c2bdeaef047df17376f62fd53037236fa2347a3ef" exitCode=0 Jul 19 15:44:16.770534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:16.770211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623273632" event={"ID":"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e","Type":"ContainerDied","Data":"7f4df7cf49ce64379a521d1c2bdeaef047df17376f62fd53037236fa2347a3ef"} Jul 19 15:44:17.915412 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:17.915386 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:17.918207 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:17.918187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwmd5\" (UniqueName: \"kubernetes.io/projected/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e-kube-api-access-bwmd5\") pod \"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e\" (UID: \"3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e\") " Jul 19 15:44:17.920204 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:17.920176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e-kube-api-access-bwmd5" (OuterVolumeSpecName: "kube-api-access-bwmd5") pod "3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" (UID: "3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e"). InnerVolumeSpecName "kube-api-access-bwmd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:17.933903 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:17.933855 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7626-140573623273632_3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e/test-curl-7626-140573623273632/0.log" Jul 19 15:44:17.958950 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:17.958919 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:17.961217 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:17.961194 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:18.019016 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.018981 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmd5\" (UniqueName: \"kubernetes.io/projected/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e-kube-api-access-bwmd5\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:18.079313 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.079237 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:18.079684 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.079670 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" containerName="test-curl-7626-140573623273632" Jul 19 15:44:18.079739 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.079686 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" containerName="test-curl-7626-140573623273632" Jul 19 15:44:18.079780 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.079771 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" containerName="test-curl-7626-140573623273632" Jul 19 15:44:18.083068 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.083044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:18.087012 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.086944 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:18.119746 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.119719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxd4\" (UniqueName: \"kubernetes.io/projected/6ad49111-ae0f-4ae4-8966-50d44e030dac-kube-api-access-fpxd4\") pod \"test-curl-7626-140573623273632\" (UID: \"6ad49111-ae0f-4ae4-8966-50d44e030dac\") " pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:18.220788 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.220752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxd4\" (UniqueName: \"kubernetes.io/projected/6ad49111-ae0f-4ae4-8966-50d44e030dac-kube-api-access-fpxd4\") pod \"test-curl-7626-140573623273632\" (UID: \"6ad49111-ae0f-4ae4-8966-50d44e030dac\") " pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:18.229004 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.228971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxd4\" (UniqueName: \"kubernetes.io/projected/6ad49111-ae0f-4ae4-8966-50d44e030dac-kube-api-access-fpxd4\") pod \"test-curl-7626-140573623273632\" (UID: \"6ad49111-ae0f-4ae4-8966-50d44e030dac\") " pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:18.394927 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.394897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:18.553009 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.552985 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:18.555014 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:18.554987 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad49111_ae0f_4ae4_8966_50d44e030dac.slice/crio-d40d692d615ecfb4360aef5d5d041bd1a05df99dfe8dc3f5f29496f84b9abc7a WatchSource:0}: Error finding container d40d692d615ecfb4360aef5d5d041bd1a05df99dfe8dc3f5f29496f84b9abc7a: Status 404 returned error can't find the container with id d40d692d615ecfb4360aef5d5d041bd1a05df99dfe8dc3f5f29496f84b9abc7a Jul 19 15:44:18.784497 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.784406 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:18.784497 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.784444 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e105e60ee98d429509586aa74b101cf45211e8f40792d146df53297471b07f" Jul 19 15:44:18.785777 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.785744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623273632" event={"ID":"6ad49111-ae0f-4ae4-8966-50d44e030dac","Type":"ContainerStarted","Data":"d40d692d615ecfb4360aef5d5d041bd1a05df99dfe8dc3f5f29496f84b9abc7a"} Jul 19 15:44:18.796672 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:18.796648 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7626-140573623273632" oldPodUID="3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" podUID="6ad49111-ae0f-4ae4-8966-50d44e030dac" Jul 19 15:44:19.663121 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:19.663087 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e" path="/var/lib/kubelet/pods/3c56c4a1-2eef-4a8f-a34a-d2ab8b52c70e/volumes" Jul 19 15:44:19.793933 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:19.793889 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ad49111-ae0f-4ae4-8966-50d44e030dac" containerID="e6b3b9d506d10f79e596a3c1386c391c867563cb351b9350cdccdaf9cec68ab8" exitCode=0 Jul 19 15:44:19.794110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:19.793933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7626-140573623273632" event={"ID":"6ad49111-ae0f-4ae4-8966-50d44e030dac","Type":"ContainerDied","Data":"e6b3b9d506d10f79e596a3c1386c391c867563cb351b9350cdccdaf9cec68ab8"} Jul 19 15:44:20.939108 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:20.939084 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:20.957439 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:20.957411 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7626-140573623273632_6ad49111-ae0f-4ae4-8966-50d44e030dac/test-curl-7626-140573623273632/0.log" Jul 19 15:44:20.979893 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:20.979863 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:20.983221 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:20.983198 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7626-140573623273632"] Jul 19 15:44:21.052646 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.052560 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpxd4\" (UniqueName: \"kubernetes.io/projected/6ad49111-ae0f-4ae4-8966-50d44e030dac-kube-api-access-fpxd4\") pod \"6ad49111-ae0f-4ae4-8966-50d44e030dac\" (UID: \"6ad49111-ae0f-4ae4-8966-50d44e030dac\") " Jul 19 15:44:21.054462 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.054438 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad49111-ae0f-4ae4-8966-50d44e030dac-kube-api-access-fpxd4" (OuterVolumeSpecName: "kube-api-access-fpxd4") pod "6ad49111-ae0f-4ae4-8966-50d44e030dac" (UID: "6ad49111-ae0f-4ae4-8966-50d44e030dac"). InnerVolumeSpecName "kube-api-access-fpxd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:21.153483 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.153450 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxd4\" (UniqueName: \"kubernetes.io/projected/6ad49111-ae0f-4ae4-8966-50d44e030dac-kube-api-access-fpxd4\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:21.169259 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.169229 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6"] Jul 19 15:44:21.169753 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.169735 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad49111-ae0f-4ae4-8966-50d44e030dac" containerName="test-curl-7626-140573623273632" Jul 19 15:44:21.169854 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.169754 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad49111-ae0f-4ae4-8966-50d44e030dac" containerName="test-curl-7626-140573623273632" Jul 19 15:44:21.169925 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.169892 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad49111-ae0f-4ae4-8966-50d44e030dac" containerName="test-curl-7626-140573623273632" Jul 19 15:44:21.173025 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.173007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:21.175124 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.175107 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:44:21.179960 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.179757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6"] Jul 19 15:44:21.254318 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.254278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2af036-abf9-4611-ab12-4bd5c63b8850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6\" (UID: \"ad2af036-abf9-4611-ab12-4bd5c63b8850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:21.355132 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.355030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2af036-abf9-4611-ab12-4bd5c63b8850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6\" (UID: \"ad2af036-abf9-4611-ab12-4bd5c63b8850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:21.355694 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.355674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2af036-abf9-4611-ab12-4bd5c63b8850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6\" (UID: \"ad2af036-abf9-4611-ab12-4bd5c63b8850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:21.485314 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.485280 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:21.646155 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.646114 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:21.663794 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.663763 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad49111-ae0f-4ae4-8966-50d44e030dac" path="/var/lib/kubelet/pods/6ad49111-ae0f-4ae4-8966-50d44e030dac/volumes" Jul 19 15:44:21.747021 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.746991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:21.806573 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.806536 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7626-140573623273632" Jul 19 15:44:21.806756 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.806576 2580 scope.go:117] "RemoveContainer" containerID="e6b3b9d506d10f79e596a3c1386c391c867563cb351b9350cdccdaf9cec68ab8" Jul 19 15:44:21.825993 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:21.825970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6"] Jul 19 15:44:21.827296 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:21.827266 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2af036_abf9_4611_ab12_4bd5c63b8850.slice/crio-6d6a10b2f955e241a5292ed20395524b77eb09cd2915079f4e897b07241297a8 WatchSource:0}: Error finding container 6d6a10b2f955e241a5292ed20395524b77eb09cd2915079f4e897b07241297a8: Status 404 returned error can't find the container with id 6d6a10b2f955e241a5292ed20395524b77eb09cd2915079f4e897b07241297a8 Jul 19 15:44:22.814383 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:22.814287 2580 generic.go:356] "Generic (PLEG): container finished" podID="ad2af036-abf9-4611-ab12-4bd5c63b8850" containerID="1a0e9257ea51975090a6b70aa55fe66356d5268784ecee68569844c69843facb" exitCode=0 Jul 19 15:44:22.814828 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:22.814371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" event={"ID":"ad2af036-abf9-4611-ab12-4bd5c63b8850","Type":"ContainerDied","Data":"1a0e9257ea51975090a6b70aa55fe66356d5268784ecee68569844c69843facb"} Jul 19 15:44:22.814828 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:22.814442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" event={"ID":"ad2af036-abf9-4611-ab12-4bd5c63b8850","Type":"ContainerStarted","Data":"6d6a10b2f955e241a5292ed20395524b77eb09cd2915079f4e897b07241297a8"} Jul 19 15:44:23.965791 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:23.965757 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:24.080372 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:24.080276 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2af036-abf9-4611-ab12-4bd5c63b8850-maas-api-service-ca\") pod \"ad2af036-abf9-4611-ab12-4bd5c63b8850\" (UID: \"ad2af036-abf9-4611-ab12-4bd5c63b8850\") " Jul 19 15:44:24.080668 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:24.080646 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2af036-abf9-4611-ab12-4bd5c63b8850-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad2af036-abf9-4611-ab12-4bd5c63b8850" (UID: "ad2af036-abf9-4611-ab12-4bd5c63b8850"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:44:24.181621 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:24.181564 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2af036-abf9-4611-ab12-4bd5c63b8850-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:24.828639 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:24.828612 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" Jul 19 15:44:24.828639 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:24.828620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6" event={"ID":"ad2af036-abf9-4611-ab12-4bd5c63b8850","Type":"ContainerDied","Data":"6d6a10b2f955e241a5292ed20395524b77eb09cd2915079f4e897b07241297a8"} Jul 19 15:44:24.828861 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:24.828669 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6a10b2f955e241a5292ed20395524b77eb09cd2915079f4e897b07241297a8" Jul 19 15:44:30.245005 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:30.244970 2580 scope.go:117] "RemoveContainer" containerID="810154223d2d8263d5b68899346d414c6c9dcc4e461e2afdd69b4757629f4143" Jul 19 15:44:31.771542 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:31.771509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:36.234155 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:36.234103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:36.244280 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:36.244247 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6"] Jul 19 15:44:36.246696 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:36.246676 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-51b4ec-dskp6"] Jul 19 15:44:36.623208 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:36.623159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:37.662315 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:37.662283 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2af036-abf9-4611-ab12-4bd5c63b8850" path="/var/lib/kubelet/pods/ad2af036-abf9-4611-ab12-4bd5c63b8850/volumes" Jul 19 15:44:42.373397 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:42.373356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:42.481332 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:42.481298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:42.684280 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:42.684195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:42.911563 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:42.911536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:43.027984 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.027901 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct"] Jul 19 15:44:43.028333 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.028320 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2af036-abf9-4611-ab12-4bd5c63b8850" containerName="revoke-keys" Jul 19 15:44:43.028375 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.028334 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2af036-abf9-4611-ab12-4bd5c63b8850" containerName="revoke-keys" Jul 19 15:44:43.028411 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.028403 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2af036-abf9-4611-ab12-4bd5c63b8850" containerName="revoke-keys" Jul 19 15:44:43.031524 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.031508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:43.033610 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.033594 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:44:43.037625 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.037603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct"] Jul 19 15:44:43.155984 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.155945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfc40e-0733-421c-af93-6e1dce427810-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c17947-gqhct\" (UID: \"60cfc40e-0733-421c-af93-6e1dce427810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:43.257431 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.257394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfc40e-0733-421c-af93-6e1dce427810-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c17947-gqhct\" (UID: \"60cfc40e-0733-421c-af93-6e1dce427810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:43.258058 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.258038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfc40e-0733-421c-af93-6e1dce427810-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c17947-gqhct\" (UID: \"60cfc40e-0733-421c-af93-6e1dce427810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:43.345644 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.345538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:43.493303 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.493270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct"] Jul 19 15:44:43.494490 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:44:43.494463 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cfc40e_0733_421c_af93_6e1dce427810.slice/crio-64c21ed738c97bb34c9a01f487cfcc2951ff8b5572f3556905f489b8f763fab4 WatchSource:0}: Error finding container 64c21ed738c97bb34c9a01f487cfcc2951ff8b5572f3556905f489b8f763fab4: Status 404 returned error can't find the container with id 64c21ed738c97bb34c9a01f487cfcc2951ff8b5572f3556905f489b8f763fab4 Jul 19 15:44:43.508200 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.508176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:43.609912 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.609884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:43.950417 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.950332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" event={"ID":"60cfc40e-0733-421c-af93-6e1dce427810","Type":"ContainerStarted","Data":"66f07c8663e8e188139360e429250f0f70d8632df4ff197c281ee5e2532fe9de"} Jul 19 15:44:43.950417 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.950385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" event={"ID":"60cfc40e-0733-421c-af93-6e1dce427810","Type":"ContainerStarted","Data":"64c21ed738c97bb34c9a01f487cfcc2951ff8b5572f3556905f489b8f763fab4"} Jul 19 15:44:43.965649 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:43.965546 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" podStartSLOduration=0.965524986 podStartE2EDuration="965.524986ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:44:43.962621132 +0000 UTC m=+2834.826895930" watchObservedRunningTime="2026-07-19 15:44:43.965524986 +0000 UTC m=+2834.829799785" Jul 19 15:44:44.957727 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:44.957695 2580 generic.go:356] "Generic (PLEG): container finished" podID="60cfc40e-0733-421c-af93-6e1dce427810" containerID="66f07c8663e8e188139360e429250f0f70d8632df4ff197c281ee5e2532fe9de" exitCode=0 Jul 19 15:44:44.958110 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:44.957785 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" event={"ID":"60cfc40e-0733-421c-af93-6e1dce427810","Type":"ContainerDied","Data":"66f07c8663e8e188139360e429250f0f70d8632df4ff197c281ee5e2532fe9de"} Jul 19 15:44:46.111014 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.110988 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:46.193048 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.193015 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfc40e-0733-421c-af93-6e1dce427810-maas-api-service-ca\") pod \"60cfc40e-0733-421c-af93-6e1dce427810\" (UID: \"60cfc40e-0733-421c-af93-6e1dce427810\") " Jul 19 15:44:46.193399 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.193373 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60cfc40e-0733-421c-af93-6e1dce427810-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60cfc40e-0733-421c-af93-6e1dce427810" (UID: "60cfc40e-0733-421c-af93-6e1dce427810"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:44:46.294565 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.294484 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60cfc40e-0733-421c-af93-6e1dce427810-maas-api-service-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\"" Jul 19 15:44:46.970312 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.970279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" event={"ID":"60cfc40e-0733-421c-af93-6e1dce427810","Type":"ContainerDied","Data":"64c21ed738c97bb34c9a01f487cfcc2951ff8b5572f3556905f489b8f763fab4"} Jul 19 15:44:46.970312 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.970311 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct" Jul 19 15:44:46.970528 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:46.970331 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64c21ed738c97bb34c9a01f487cfcc2951ff8b5572f3556905f489b8f763fab4" Jul 19 15:44:53.419224 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:53.419194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:58.093820 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:58.093784 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct"] Jul 19 15:44:58.096044 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:58.096016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:58.096936 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:58.096917 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c17947-gqhct"] Jul 19 15:44:58.484911 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:58.484873 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:44:59.662477 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:44:59.662445 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60cfc40e-0733-421c-af93-6e1dce427810" path="/var/lib/kubelet/pods/60cfc40e-0733-421c-af93-6e1dce427810/volumes" Jul 19 15:45:04.357308 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:04.357275 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:04.505844 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:04.505815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:04.716196 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:04.716115 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:04.887626 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:04.887568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:10.677849 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:10.677820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpqw4_e2627744-13cd-4b05-881c-b3cf99960928/manager/0.log" Jul 19 15:45:10.797424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:10.797395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-87vjj_877b8a5c-8140-4e48-b08b-3c2900ab4815/manager/0.log" Jul 19 15:45:11.053058 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:11.052980 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-57bzz_a3210330-e30d-4248-8a9f-5dc79531d1ef/server/0.log" Jul 19 15:45:11.169101 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:11.169072 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rp424_0c8f1435-8393-46f7-b8df-7a696a2863d6/manager/0.log" Jul 19 15:45:11.536079 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:11.536050 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-x6vbt_af59f2f3-37b4-4c1a-8c05-ed811336b1ff/copy-manifests/0.log" Jul 19 15:45:11.543791 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:11.543763 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-x6vbt_af59f2f3-37b4-4c1a-8c05-ed811336b1ff/manager/0.log" Jul 19 15:45:11.897599 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:11.897554 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6db48d95f-j8pjv_521464ff-46c0-40f1-bf39-a41c6d5d14cd/maas-api/0.log" Jul 19 15:45:12.363150 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:12.363123 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4f5nk_860e4b9e-5c4f-4d58-82db-0bab17c14c92/postgres/0.log" Jul 19 15:45:13.133854 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.133827 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t_338e367b-0cee-411c-9f74-e26eef0dc377/util/0.log" Jul 19 15:45:13.140250 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.140229 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t_338e367b-0cee-411c-9f74-e26eef0dc377/pull/0.log" Jul 19 15:45:13.147053 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.147033 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t_338e367b-0cee-411c-9f74-e26eef0dc377/extract/0.log" Jul 19 15:45:13.262199 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.262172 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q_249af867-feba-439f-b860-da70ac3387d2/util/0.log" Jul 19 15:45:13.268708 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.268686 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q_249af867-feba-439f-b860-da70ac3387d2/pull/0.log" Jul 19 15:45:13.274941 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.274920 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q_249af867-feba-439f-b860-da70ac3387d2/extract/0.log" Jul 19 15:45:13.385607 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.385496 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992_63fd2a43-9ad8-4b8a-b023-687f4bfff9cd/util/0.log" Jul 19 15:45:13.392618 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.392598 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992_63fd2a43-9ad8-4b8a-b023-687f4bfff9cd/pull/0.log" Jul 19 15:45:13.398128 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.398111 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992_63fd2a43-9ad8-4b8a-b023-687f4bfff9cd/extract/0.log" Jul 19 15:45:13.508227 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.508199 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb_b5cea9a8-a913-4a95-bf98-6c42dba66e35/extract/0.log" Jul 19 15:45:13.514008 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.513989 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb_b5cea9a8-a913-4a95-bf98-6c42dba66e35/util/0.log" Jul 19 15:45:13.519655 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.519637 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb_b5cea9a8-a913-4a95-bf98-6c42dba66e35/pull/0.log" Jul 19 15:45:13.754158 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.754081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-55mt5_8ea3c2dd-c626-48ee-aa71-a35912a689d5/manager/0.log" Jul 19 15:45:13.866475 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.866445 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n9qzk_7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1/manager/0.log" Jul 19 15:45:13.982206 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:13.982180 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zbbb7_7cc7aeb2-3edc-4449-bcae-d94bb77456da/kuadrant-console-plugin/0.log" Jul 19 15:45:14.105428 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:14.105355 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-b94l7_2350647f-1aae-40b7-885d-fac03cdee35c/registry-server/0.log" Jul 19 15:45:14.459164 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:14.459128 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rb9ml_0f7f1a0d-78ea-431d-bc88-cfca9e98a75a/manager/0.log" Jul 19 15:45:14.810043 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:14.809961 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4g6hxh_0b734a60-94d8-4205-8d3f-f71aef853f9e/istio-proxy/0.log" Jul 19 15:45:15.051517 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:15.051487 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-94xbq_76cfdf0d-eef2-4908-bf8f-3403c4aec4b7/kube-auth-proxy/0.log" Jul 19 15:45:15.285908 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:15.285864 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/1.log" Jul 19 15:45:15.522973 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:15.522942 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-czp9q_df93acff-04ce-4151-b643-68ce4023c5b0/payload-processing/0.log" Jul 19 15:45:15.633651 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:15.633627 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75784d96f4-xmj78_17ac597b-4a6e-4a0f-a36e-5914098f469b/router/0.log" Jul 19 15:45:15.971538 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:15.971462 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qwjsn_991ded96-c855-43d8-a32f-c53290de37ec/main/0.log" Jul 19 15:45:16.309423 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:16.309344 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f44qj6_2d871e4c-674f-43fb-8c9e-5d57913f0ee6/main/0.log" Jul 19 15:45:16.425543 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:16.425505 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qcc9s_a18aabae-f4b9-4e39-a984-e9fd550685ed/main/0.log" Jul 19 15:45:21.925093 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:21.925010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:22.021146 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:22.021113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:23.849121 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:23.849085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:23.904027 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:23.903993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:24.741599 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:24.741558 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rgb9j_63cfb73d-a2d1-41a6-8911-03772f734c2d/global-pull-secret-syncer/0.log" Jul 19 15:45:24.783416 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:24.783383 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" Jul 19 15:45:24.885239 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:24.885207 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-163.ec2.internal_b4a6110787fb36561c0436ce6584028c/haproxy/0.log" Jul 19 15:45:29.341086 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.341058 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t_338e367b-0cee-411c-9f74-e26eef0dc377/extract/0.log" Jul 19 15:45:29.377127 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.377095 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t_338e367b-0cee-411c-9f74-e26eef0dc377/util/0.log" Jul 19 15:45:29.398674 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.398644 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cxj6t_338e367b-0cee-411c-9f74-e26eef0dc377/pull/0.log" Jul 19 15:45:29.423133 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.423106 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q_249af867-feba-439f-b860-da70ac3387d2/extract/0.log" Jul 19 15:45:29.445606 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.445556 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q_249af867-feba-439f-b860-da70ac3387d2/util/0.log" Jul 19 15:45:29.462019 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.461991 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0vf57q_249af867-feba-439f-b860-da70ac3387d2/pull/0.log" Jul 19 15:45:29.484826 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.484791 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992_63fd2a43-9ad8-4b8a-b023-687f4bfff9cd/extract/0.log" Jul 19 15:45:29.510837 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.510809 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992_63fd2a43-9ad8-4b8a-b023-687f4bfff9cd/util/0.log" Jul 19 15:45:29.530080 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.530054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rl992_63fd2a43-9ad8-4b8a-b023-687f4bfff9cd/pull/0.log" Jul 19 15:45:29.555798 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.555773 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb_b5cea9a8-a913-4a95-bf98-6c42dba66e35/extract/0.log" Jul 19 15:45:29.577442 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.577418 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb_b5cea9a8-a913-4a95-bf98-6c42dba66e35/util/0.log" Jul 19 15:45:29.599378 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:29.599302 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fdwmb_b5cea9a8-a913-4a95-bf98-6c42dba66e35/pull/0.log" Jul 19 15:45:30.112925 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:30.112895 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-55mt5_8ea3c2dd-c626-48ee-aa71-a35912a689d5/manager/0.log" Jul 19 15:45:30.137353 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:30.137323 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n9qzk_7f3cb68d-15b1-4a19-a72f-e12b74c2b2c1/manager/0.log" Jul 19 15:45:30.164844 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:30.164817 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zbbb7_7cc7aeb2-3edc-4449-bcae-d94bb77456da/kuadrant-console-plugin/0.log" Jul 19 15:45:30.202789 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:30.202759 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-b94l7_2350647f-1aae-40b7-885d-fac03cdee35c/registry-server/0.log" Jul 19 15:45:30.441181 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:30.441090 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rb9ml_0f7f1a0d-78ea-431d-bc88-cfca9e98a75a/manager/0.log" Jul 19 15:45:32.057090 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.057059 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-skx69_08f073a5-104f-4805-af4f-afd33d9c109b/cluster-monitoring-operator/0.log" Jul 19 15:45:32.252523 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.252496 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdk76_6b3b9ec8-6733-48c2-8ec8-97b468132c01/node-exporter/0.log" Jul 19 15:45:32.277158 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.277133 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdk76_6b3b9ec8-6733-48c2-8ec8-97b468132c01/kube-rbac-proxy/0.log" Jul 19 15:45:32.301738 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.301709 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdk76_6b3b9ec8-6733-48c2-8ec8-97b468132c01/init-textfile/0.log" Jul 19 15:45:32.858078 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.858049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79f8844fd6-7xgrs_24af6a87-1591-48d5-9f1e-ab6f6f6634a4/telemeter-client/0.log" Jul 19 15:45:32.880504 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.880474 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79f8844fd6-7xgrs_24af6a87-1591-48d5-9f1e-ab6f6f6634a4/reload/0.log" Jul 19 15:45:32.905924 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:32.905899 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79f8844fd6-7xgrs_24af6a87-1591-48d5-9f1e-ab6f6f6634a4/kube-rbac-proxy/0.log" Jul 19 15:45:34.390397 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:34.390368 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-jwjbv_508b0e64-758e-4cc0-aa21-eb791dcc95f8/networking-console-plugin/0.log" Jul 19 15:45:34.978168 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:34.978137 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/1.log" Jul 19 15:45:34.982231 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:34.982212 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-f662r_f03f5080-5b9d-411a-b5f4-aab8f1f5e5c9/console-operator/2.log" Jul 19 15:45:35.078386 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.078350 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq"] Jul 19 15:45:35.078974 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.078952 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60cfc40e-0733-421c-af93-6e1dce427810" containerName="revoke-keys" Jul 19 15:45:35.078974 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.078973 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60cfc40e-0733-421c-af93-6e1dce427810" containerName="revoke-keys" Jul 19 15:45:35.079126 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.079068 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60cfc40e-0733-421c-af93-6e1dce427810" containerName="revoke-keys" Jul 19 15:45:35.083228 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.083212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.085424 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.085405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h7n64\"/\"openshift-service-ca.crt\"" Jul 19 15:45:35.085574 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.085556 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h7n64\"/\"kube-root-ca.crt\"" Jul 19 15:45:35.085658 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.085647 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h7n64\"/\"default-dockercfg-7hpkc\"" Jul 19 15:45:35.091840 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.091817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq"] Jul 19 15:45:35.262441 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.262351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7n4\" (UniqueName: \"kubernetes.io/projected/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-kube-api-access-bs7n4\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.262441 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.262401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-proc\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.262685 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.262470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-podres\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.262685 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.262505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-lib-modules\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.262685 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.262526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-sys\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363687 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-podres\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-lib-modules\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-sys\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7n4\" (UniqueName: \"kubernetes.io/projected/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-kube-api-access-bs7n4\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-proc\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-podres\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.363880 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-sys\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.364083 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-proc\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.364083 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.363906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-lib-modules\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.372678 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.372640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7n4\" (UniqueName: \"kubernetes.io/projected/3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9-kube-api-access-bs7n4\") pod \"perf-node-gather-daemonset-8zqhq\" (UID: \"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.393590 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.393561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:35.531308 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.531278 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq"] Jul 19 15:45:35.532985 ip-10-0-140-163 kubenswrapper[2580]: W0719 15:45:35.532960 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e9f6dd8_d2a5_4c5e_b745_3d0cca4fb4b9.slice/crio-cbed8dd244ace98100010a810ed5502ed69c9cb865bfd45c5b1ee1f6ba8623a5 WatchSource:0}: Error finding container cbed8dd244ace98100010a810ed5502ed69c9cb865bfd45c5b1ee1f6ba8623a5: Status 404 returned error can't find the container with id cbed8dd244ace98100010a810ed5502ed69c9cb865bfd45c5b1ee1f6ba8623a5 Jul 19 15:45:35.633273 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:35.633244 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-6chsd_71ca793f-d8d5-49bf-a27c-6c821e39fcea/download-server/0.log" Jul 19 15:45:36.256049 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:36.256022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-k7rbl_8fbbbb06-6826-4239-9aea-1909f0723059/volume-data-source-validator/0.log" Jul 19 15:45:36.261211 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:36.261183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" event={"ID":"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9","Type":"ContainerStarted","Data":"d1012808de9ad1ff861200276e0bab605c4dde0eee528845cdec6dec104c0e55"} Jul 19 15:45:36.261359 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:36.261234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" event={"ID":"3e9f6dd8-d2a5-4c5e-b745-3d0cca4fb4b9","Type":"ContainerStarted","Data":"cbed8dd244ace98100010a810ed5502ed69c9cb865bfd45c5b1ee1f6ba8623a5"} Jul 19 15:45:36.261359 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:36.261336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:36.275935 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:36.275883 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" podStartSLOduration=1.275867854 podStartE2EDuration="1.275867854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 15:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:45:36.274768624 +0000 UTC m=+2887.139043433" watchObservedRunningTime="2026-07-19 15:45:36.275867854 +0000 UTC m=+2887.140142651" Jul 19 15:45:37.354919 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:37.354894 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ttnr8_b6c64568-904b-478f-a9d8-a8b9151a0748/dns/0.log" Jul 19 15:45:37.373057 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:37.373032 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ttnr8_b6c64568-904b-478f-a9d8-a8b9151a0748/kube-rbac-proxy/0.log" Jul 19 15:45:37.394012 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:37.393982 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7k7rd_c1e81ea0-1dea-466e-a980-176b3a996a7d/dns-node-resolver/0.log" Jul 19 15:45:37.977534 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:37.977506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mscg5_754bd0cf-54de-4f3f-b513-501ec601ec40/node-ca/0.log" Jul 19 15:45:38.912989 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:38.912949 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4g6hxh_0b734a60-94d8-4205-8d3f-f71aef853f9e/istio-proxy/0.log" Jul 19 15:45:39.051120 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:39.051090 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-94xbq_76cfdf0d-eef2-4908-bf8f-3403c4aec4b7/kube-auth-proxy/0.log" Jul 19 15:45:39.104249 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:39.104215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/0.log" Jul 19 15:45:39.129955 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:39.129918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/1.log" Jul 19 15:45:39.183118 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:39.183037 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-czp9q_df93acff-04ce-4151-b643-68ce4023c5b0/payload-processing/0.log" Jul 19 15:45:39.203877 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:39.203850 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75784d96f4-xmj78_17ac597b-4a6e-4a0f-a36e-5914098f469b/router/0.log" Jul 19 15:45:39.910293 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:39.910264 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ndgfw_83275949-83ec-4fb7-9ca7-7dd6ebe2ce10/serve-healthcheck-canary/0.log" Jul 19 15:45:40.361023 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:40.360994 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-sb2j7_bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e/insights-operator/0.log" Jul 19 15:45:40.362004 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:40.361983 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-sb2j7_bcdf2c22-83b3-4475-98b0-6d1dcdaf8a7e/insights-operator/1.log" Jul 19 15:45:40.503915 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:40.503886 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw4dv_935ad60d-8783-4dee-bb2c-692896d4e497/kube-rbac-proxy/0.log" Jul 19 15:45:40.523313 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:40.523283 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw4dv_935ad60d-8783-4dee-bb2c-692896d4e497/exporter/0.log" Jul 19 15:45:40.541742 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:40.541722 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw4dv_935ad60d-8783-4dee-bb2c-692896d4e497/extractor/0.log" Jul 19 15:45:42.276415 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:42.276390 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-8zqhq" Jul 19 15:45:42.582208 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:42.582129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpqw4_e2627744-13cd-4b05-881c-b3cf99960928/manager/0.log" Jul 19 15:45:42.769398 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:42.769366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-87vjj_877b8a5c-8140-4e48-b08b-3c2900ab4815/manager/0.log" Jul 19 15:45:42.905478 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:42.905452 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-57bzz_a3210330-e30d-4248-8a9f-5dc79531d1ef/server/0.log" Jul 19 15:45:42.969253 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:42.969221 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rp424_0c8f1435-8393-46f7-b8df-7a696a2863d6/manager/0.log" Jul 19 15:45:43.147505 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:43.147471 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-x6vbt_af59f2f3-37b4-4c1a-8c05-ed811336b1ff/manager/0.log" Jul 19 15:45:43.170611 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:43.170521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-x6vbt_af59f2f3-37b4-4c1a-8c05-ed811336b1ff/copy-manifests/0.log" Jul 19 15:45:44.681760 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:44.681727 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mx6jp_266b3d70-24cc-469e-8906-d6822c6beaa7/konnectivity-agent/0.log" limit=1024 Jul 19 15:45:45.865369 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:45.865339 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4g6hxh_0b734a60-94d8-4205-8d3f-f71aef853f9e/istio-proxy/0.log" Jul 19 15:45:45.999277 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:45.999245 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-94xbq_76cfdf0d-eef2-4908-bf8f-3403c4aec4b7/kube-auth-proxy/0.log" Jul 19 15:45:46.030531 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:46.030497 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/0.log" Jul 19 15:45:46.062377 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:46.062336 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/1.log" Jul 19 15:45:46.101569 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:46.101542 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-czp9q_df93acff-04ce-4151-b643-68ce4023c5b0/payload-processing/0.log" Jul 19 15:45:46.113533 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:46.113512 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75784d96f4-xmj78_17ac597b-4a6e-4a0f-a36e-5914098f469b/router/0.log" Jul 19 15:45:47.350704 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:47.350672 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4g6hxh_0b734a60-94d8-4205-8d3f-f71aef853f9e/istio-proxy/0.log" Jul 19 15:45:47.482549 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:47.482523 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647d897b75-94xbq_76cfdf0d-eef2-4908-bf8f-3403c4aec4b7/kube-auth-proxy/0.log" Jul 19 15:45:47.514555 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:47.514526 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/0.log" Jul 19 15:45:47.545349 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:47.545320 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-zsgrq_f09ac2ee-ebf4-494e-a842-91be16d9bc6a/istio-proxy/1.log" Jul 19 15:45:47.591159 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:47.591131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-czp9q_df93acff-04ce-4151-b643-68ce4023c5b0/payload-processing/0.log" Jul 19 15:45:47.608190 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:47.608162 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75784d96f4-xmj78_17ac597b-4a6e-4a0f-a36e-5914098f469b/router/0.log" Jul 19 15:45:50.143666 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:50.143622 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-b864s_125a2380-9e33-47d6-926a-9af477e93234/kube-storage-version-migrator-operator/1.log" Jul 19 15:45:50.144717 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:50.144698 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-b864s_125a2380-9e33-47d6-926a-9af477e93234/kube-storage-version-migrator-operator/0.log" Jul 19 15:45:51.216628 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.216575 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/kube-multus-additional-cni-plugins/0.log" Jul 19 15:45:51.235635 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.235604 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/egress-router-binary-copy/0.log" Jul 19 15:45:51.253626 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.253599 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/cni-plugins/0.log" Jul 19 15:45:51.271594 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.271567 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/bond-cni-plugin/0.log" Jul 19 15:45:51.289436 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.289408 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/routeoverride-cni/0.log" Jul 19 15:45:51.308300 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.308280 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/whereabouts-cni-bincopy/0.log" Jul 19 15:45:51.327568 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.327544 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hcjp9_c34f641b-e476-47c0-9dce-506e0d8b2c36/whereabouts-cni/0.log" Jul 19 15:45:51.507661 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.507566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c689m_b035e097-e89a-46e2-ac9e-83543a280851/kube-multus/0.log" Jul 19 15:45:51.635346 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.635313 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xpbn5_a4142486-9667-4b0c-9304-d6edc4c4d09b/network-metrics-daemon/0.log" Jul 19 15:45:51.652230 ip-10-0-140-163 kubenswrapper[2580]: I0719 15:45:51.652203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xpbn5_a4142486-9667-4b0c-9304-d6edc4c4d09b/kube-rbac-proxy/0.log"